Jacso, P. (2010).
“Eigenfactor and Article Influence Scores
in the Journal Citation Reports
Print version to be published in
Online Information Review 2010 Vol. 34, No. 2, pp. 339-348
This sequel to the earlier
testing and evaluation of the 5-year Journal Impact Factor (JIF-5) in the
enhanced second edition of the Journal Citation Reports for 2007 (JCR-2007),
released in January, 2009, and for the 2008 edition, released in July, 2009,
assesses and compares the impact on the ranking of journals by two other
performance indicators. Both the Eigenfactor Score (EFS) and the Article Influence Score (AFS) use a
5-year target window in the algorithm to quantify the scholarly impact at the
overall journal level and at the article level, respectively. It is examined
how the rank positions of 52 library and information science journals change
when the set of journals are ranked by the Eigenfactor metrics vis-à-vis the
JIF-5 indicator.
Background
The JCR has been available
for nearly 40 years if we count the print, microfiche, CD-ROM and online
formats. The JCR and the traditional Journal Impact Factor based on a two year
target window (JIF-2) have been
controversial but also widely (if not always wisely) used directly or as a
proxy for a variety of purposes, ranging
from library collection development to evaluating research quality of
college departments, institutions and countries, to making decisions in tenure,
promotion and grant applications.
There have been thousands of
papers published about the pros and cons of the journal impact factor in
general, and the results of its use in ranking journals in specific disciplines
and application areas. It is quite telling that the National Library of
Medicine added – in addition to the long existing, broader subject heading,
Bibliometrics - the narrower term, Journal Impact Factor as a Medical Subject
Heading (MeSH) in 2008, and in two years it was assigned to 288 MEDLINE
records. The free text search in the abstracting/indexing records for journal*
AND impact factor* (as an exact phrase
for the latter, and accommodating both the singular and plural formats of both
terms to retrieve journal impact factors, impact factor of journals, and other
variations) finds more than 1,000 records, obviously from medical and life
sciences journals alone.
There is a very high
redundancy in the majority of papers, parroting the same old, same old reasons
for the pros and the cons that were described, admitted and well explained
decades ago by Eugene Garfield and his fellow scientist, Irving Sher (Garfield
and Sher, 1963, Garfield, 1972). Fortunately, the most competent, real experts
in bibliometrics and scientometrics, and informetrics, also published
enlightening papers (Glanzel, 2009, Leydesdorff, 2008, Moed and van Leeuwen,
1995, Nisonger, 1994, 2000 and 2004, Pendlebury, 2009, Rousseau, 2001 and
2005), or summarized the essential issues and positions in their objective and
informing reviews of the broader literature (Bar-Ilan, 2008, Wilson, 1999).
On a personal level, I have
been using the JCR from my early years as a practitioner and later as an
academic with great appreciation (Jacso, 2005), and with reservation (Jacso,
2000 and 2001). I found the introduction of the JIF-5 indicator an important
and much needed step forward (Jacso, 2009).
Amidst the many unusually
extremist standpoints, the recent case study (Linda Butler, 2008) showed the
refreshingly rational attitude of
adopting a balanced approach in using
bibliometric indicators. This goes hand in hand with Stevan Harnad’s principle
of validating research performance metrics against peer rankings (Harnad,
2008). The more the better, I would add, especially as the nationwide projects
of the research assessment of universities and colleges through the prism of
faculty publications output in journals ranked by peers into 4 tiers, are
getting in full gear. This is particularly so in the UK and Australia, where
Charles Oppenheim’s original idea, presented first about 15 years ago, of using
citation counts in the Research Assessment Exercise (Oppenheim, 1996), are
embraced, but not in the manner of an exclusive - either this or that but not
both- Boolean XOR operation. It must be
also borne in mind that administrators might be too eager to look up, accept
and use just the indicators reported,
without understanding their limitations, and face the situation that Gary
Gorman described in his editorial “They can’t read, but they sure can count”,
about the flawed rules and malpractice in assessing the performance of
researchers (Gorman, 2008). Neither is it likely that all decision-makers would
consistently apply the standards of good practice in interpreting the results of bibliometric data (Bornmann et
al.), or would be fully aware of the metadata mega mess in Google Scholar,
coupled with a very loose citation matching algorithm (Jacso, 2010), let alone
of other, less obvious database content
and software limitations that can distort
bibliometric measures calculated from cited reference enhanced databases
(Jacso, 2008a, Jacso, 2008b, Jacso, 2006).
The incorporation of the
Eigenfactor metrics
Apparently, the launch of
the open access service at the eigenfactor.org site in 2007 gave the impetus to
Thomson (now Thomson-Reuters) to enhance the JCR. It was an odd situation. The
developers of the free service at eigenfactor.org, Associate Professor Carl
Bergstrom and his team, applied a smart idea (borrowed from the principle of
Google’s PageRank). While they used the underlying data collected and processed
by Eugene Garfield’s company, they strongly criticized the JCR itself. Educated
for a career as an intellectual property lawyer, it was not difficult to
perceive the service in its initial year as a derivative work, getting the
precious data from a proprietary source, that is not like the phone listings in the White
Pages, which are not protected (at least in the U.S.) by copyright law.
I can only presume and
speculate that after many rounds of discussions (and Thomson-Reuters’ decision
to enhance by a 5-year version
(JIF-5) the traditional JIF-2 scores)
the parties must have come to an agreement. This is beneficial for the user
community for several reasons. From the second edition of JCR-2007 onward the
JCR will be enhanced by the JIF-5 (which
is a better impact measure than JIF-2 that relies on a too narrow target
window, considering the fact that in most disciplinary areas the citation
zenith is reached 3-4 years after publication). The JCR will also incorporate
the two Eigenfactor metrics that provide an alternative measure for the same
five year period. In turn, the developers of the eigenfactor.org site may keep
on using the JCR’s proprietary base datasets for the 1995-2007 editions of more
than 10 million master records and about 200-250 million references, and -among others- would explicitly and
prominently display the note that Thomson-Reuters provided the data used for
the Eigenfactor service.
Whatever the details were,
the subscriber community now can enjoy the advantage of having three additional
journal indicators integrated into the JCR, and anyone can use free of charge
the Eigenfactor service. While it is more restricted in one sense (by not
showing the very informative and precious details of the JCR records), it is
more comprehensive - by virtue of computing additional metrics, and presenting
additional scientometric information through illuminating maps and motion charts
about the citation network of sharing scientific information through references
in publications. It is an impressive, very well designed system that deserves a
dedicated review of its own.
There is an excellent
section at the web site http://www.eigenfactor.org/methods.htm
that describes and illustrates the eigenfactor methodology. A paper published
in College & Research Libraries News (Bergstrom, 2007) also provides
additional background. So it suffices to
state here that the Eigenfactor score (EFS) is a size-dependent measure of the
overall prestige/importance of the journals from the perspective of the
advanced researcher community (but not necessarily for the college libraries
with only undergraduate and some graduate programs), while the Article
Influence Score (AIS) is a normalized score produced by dividing the EFS by the
number of papers published in the journal, indicating the average impact of each of the journal’s articles.
AIS is the indicator which
is comparable -in principle - to the JIF-5. It is comparable only in principle
because while the JIF scores treat all citations received of equal value, the
Eigenfactor scores reflect the prestige of the citing journal, i.e. the
algorithm makes a distinction between citations received from a high ranking,
important journal and a lower ranking
journal.
This was the principle
behind Google’s PageRank, where a Web
page or site got positioned in ranking the search results based not merely on the
number of incoming links, but also on
the status/prestige of the linking sites based on the PageRank scores of those
linking sites. This is a recursively calculated value that remains permanent until the next year’s
edition.
Taking the risk of sounding
blasphemous, it is the scholarly analogue of the Hollywood practice, where it
matters not only how many people attended one’s party, or how many air-kisses
the host received, but also how many of them were celebrities, and how big celebrities they have been in the invisible
college of the Hollywood glitterati where these scores are very up-to-date, and
much better known than the influence scores of journals by researchers.
Integrated presentation of
the Eigenfactor and JIF metrics
The new hub page of the JCR
shows all the formerly displayed metada
and metrics along with the new metrics in a matrix format. The JIF and
Eigenfactor metrics are in different scales so they are difficult to compare at
a glance, especially because of the tiny
values in the Eigenfactor Scores (EFS)
with 5 decimal points precision.
Figure 1. Excerpt of the summary hub page
of JCR-2007 enhanced by new indicators – ranked by the 5-year Journal Impact
Factor (JIF-5) values
By removing the ISSN column
from this page (where it has no relevance), and hiding the columns for the
Immediacy Index, the number of articles in the census year, and the Cited Half-life values, the really important
rank positions of the journals by JIF-5, AFS and AIS could be shown.
A mock-up of a more helpful
hub page was created by downloading the data into a spreadsheet to do many
calculations with the indicators, and to illustrate the feasibility of a screen
that provides an at-a-glance view showing the most critical impact indicators
for the journals. I used only 52 of the 56 journals from the Information
and Library Science (ILS) category in
the second edition of JCR-2007 (which added the Eigenfactor metrics), because
four journals did not have a JIF-5 value, as they have not been covered for
five years by Thomson Reuters (or were not even in existence for that many
years). This layout and content immediately gives a sense about the rank
position differences by the various impact measures.
Figure 2. Rank position differences by key journal
indicators within JCR-2007
I converted parts of the
above table into bump charts as they make the rank order differences more
readily apparent. The print version of this paper could accommodate only an
excerpt of the bump chart, this enhanced digital version makes it possible to
show the positions of all the qualifying 52 journals of the 56 listed in
JCR-2007.
Changes between rank
positions by JIF versus EFS
There is a dramatic negative
change only in the rank position of two of the 52 serials between the rankings
based on JIF-5 versus EFS. Although JIF-5 and EFS are not the primary
comparators as they do not measure the same impact as discussed earlier, the
differences are striking for the Annual Review of Information Science and
Technology (ARIST), and the Information Systems Journal. Their 20 and 21 rank
positions demotion by EFS may be explained – if not justified- by the typically
low number of items published per year (about 15 and 18) in these two serials.
Both Restaurator and Law
Library Journal have a loss of 16 positions. Government Information Quarterly
and Knowledge Organization drop by 14 and 11 positions, Interlending and
Document Supply by 10. In all cases – except for Knowledge Organization- the
combination of low productivity and high (45%) or very high (67%, 81% and 82%)
self-citation rate is the reason for the significant changes.
Figure 3. Bump chart for
better visualization of rank position changes by JIF-5 versus EFS
The opposite is true for the
Journal of the American Society of Information Science & Technology, which
is 8 positions higher by Eigenfactor than by JIF-5. It is not only highly cited
(by many high ranking journals both in the Science and Social Sciences
categories) but is also very productive, and productivity has high impact on
the size dependent EFS. The same is true for Scientometrics which rises 6
positions by EFS.
In the bottom 20 stratum by
JIF-5, Library Journal and the Scientist move up from their very low positions
of 48 and 50, to the 15th and the 20th position when
ranked by EFS. This is obviously because of their highest productivity among
the 52 journals. Online moves up by 9 positions splitting the 34th
position with Social Science Information which rose slightly.
Changes between rank
positions by JIF versus AIS
Some of the changes in rank positions between JIF-5 and
AIS may seem surprising, as the AIS is
normalized by volume of papers published – just as JIF-5 is. The 7 position
drop in the rank of JMLA is somewhat enigmatic. The downward position change of
Scientometrics can be explained by the fact that it is ranked 6th by self-citation rate, but
this does not explain the same degree of demotion of Information Systems
Journal which has a modest self-citation rate of 16%.
The very high self citation
rate penalize the journals that were
demoted also by EFS. Inversely, the top 6 journals by JIF-5 retain their top
positions for a combination of low or very low self-citation rates and citations
received from many high ranking journals (outside of the ILS category). MIS Quarterly, Information Systems Research,
ARIST, Information and Management, Journal of Information Science, and Online
Information Review keep their same rank
positions by JIF-5 and AIS.
There are similar changes
among the bottom 20 journals ranked by AIS, and none of the journals keep the
same positions, except for Zeitschrift für Bibliothekswesen und Bibliographie
which ranks as the last among the 52 journals by every impact factor and
Eigenfactor measures in JCR-2007. The degree of convergence is significantly
lower by the two rankings in this stratum. Once again, this digital
version shows the rank position changes
of all the 52 journals at http://www.jacso.info/jcr-eigenfactor.html also for the JIF-5 and AIS ranks.
The rankings by the three
different methods yield very different rank positions that rank correlation
coefficients would not reveal as well as the display of the rank positions in
the same row, and especially through the bump charts. It deserves further
investigation why the supposedly similar measures are not as similar as
expected even in the top 20 group where ranks usually converge much more than
in the bottom 20 groups. Minor position changes come with the limitations of
the ordinal rank numbers, and are common in rankings based on opinion of peers.
Those cannot be explored let alone recreated, but with citations-based ranking
this can be done. As the 2007 JCR set remains in a “frozen” state, further
tests can be made.
Figure 4. Bump chart for
better visualization of rank positions changes by JIF-5 versus AIS
Unfortunately, the
information rich detail pages of JCR (about citations received and given by the
journals, their exact share, distribution rate per years) do not provide
information about the self-citation rate for the 5-year time span as readily as
for the 2-year window. This is important as the major reason for the
differences for the most divergent rank positions is the treatment of self-citations by JIS-5 and the
Eigenfactor methods. The former includes self-citations, the latter excludes
them. This has made the greatest difference in the rank position of many of the
sample journals, especially in the case of Law Library Journal that had an excessively high, more than 80%
self-citation rate. I started to calculate self-citation rates for the 52 LIS
journals, but it is a tedious process. After having completed that process, it
is easy to calculate the JIF-5 scores
without self-citations, and compare them with the EFS and AIS
indicators.
Given the same raw dataset, this would give an opportunity to
focus on the other big difference between the JCR and Eigenfactor scores, the
prestige of the journals giving the
citations.
Looking up the many extra
features at the Eigenfactor site would reveal –among others- an additional
bibliometric measure, that happens to be called
Impact Factor. It is expressed in the much more readily comprehensible
and comparable and finer percentile figures, in order to validate the
convergence between the AIS and JIF-5 indicators. It will come as a sequel to
this preliminary exploration.
The smartest approach is to
use and compare the variety of bibliometric indicators not by themselves alone,
but as a tool to inform the peers of the disciplinary areas, who compile the
great variety of journal league lists. This can
make their decisions better, and help in reaching better consensus in
determining the best journals of the disciplines for different purposes and
target audiences.
References
Bar-Ilan,
J. (2008), “Informetrics at the beginning of the 21st century – A
review”, Journal of Informetrics, Vol. 2 No. 1, pp. 1-52.
Bergstrom, C.T. (2007), “Eigenfactor:
Measuring the value and prestige of scholarly journals”, College & Research Libraries News, Vol. 68 No 5, pp. 314-316.
Bornmann,
L., Mutz, R., Neuhaus, C. and Daniel, H.D. (2008), “Citation counts for research
evaluation: Standards of good practice for analyzing bibliometric data and
presenting and interpreting results”, Ethics
in Science and Environmental Politics, Vol. 8 No. 1, pp. 93-102.
Garfield,
E. (1972), “Citation analysis as a tool in journal evaluation”. Science, Volume 178 No. 60, pp. 471-479.
Garfield,
E. and Sher, I. (1963), “New factors in the evaluation of scientific literature
through citation indexing”, American
Documentation, Vol. 14, No. 3, pp. 195-201.
Glänzel, W. (2009), “The
multi-dimensionality of journal impact” Scientometrics, Vol. 78 No. 2,
pp. 355-374.
Gorman, G.E. (2008), “They can't read, but they sure can
count - Flawed rules of the journal rankings game”, Online Information
Review, Vol. 32 No. 6, pp.
705-708.
Harnad, S.
(2008), “Validating research performance metrics against peer rankings”, Ethics
in Science and Environmental Politics, Vol. 8 No. 1, pp. 103-107.
Jacso, P.
(2000), “The number game”, Online Information Review, Vol. 24 No. 2, pp.
180-183.
Jacso, P.
(2001), “A deficiency in the algorithm for calculating the impact factor of
scholarly journals: The journal impact factor”, Cortex, Vol. 37 No.
4, pp. 590-594.
Jacso, P.
(2005), “Journal Citation Reports”, Peter’s Digital Reference Shelf available
at http://www.jacso.info/gale/jcr/jcr.htm
Jacso, P.
(2006). “Deflated, inflated, and phantom citation counts”, Online Information Review, Vol. 30 No. 3, pp. 297-309.
Jacso, P.
(2008a). “The plausibility of computing the h-index of scholarly productivity
and impact using reference enhanced databases”, Online Information Review, Vol. 32 No. 2, pp. 266-283.
Jacso, P.
(2008b). “Testing the calculation of a realistic h-index in Google Scholar,
Scopus, and Web of Science for F.W. Lancaster”, Library Trends, Vol. 56 No. 4, pp. 784-815.
Jacso, P. (2009), “Five-year Impact Factor Data in the
Journal Citation Reports”, Online Information Review, Vol. 33 No. 3, pp. 603-614.
Jacso, P.
(2010). “Metadata mega mess in Google Scholar”, Online Information Review, Vol. 34 No. 1, pp. 175-191
Leydesdorff,
L. (2008), “Caveats for the use of citation indicators in research and journal
evaluations”, Journal of the American Society for Information Science and
Technology, Vol. 59 No. 2, pp. 278-287.
Moed, H.F.
and van Leeuwen, T.H. (1995). “Improving the accuracy of [the] Institute for
Scientific Information’s journal impact factors”, Journal of the American
Society for Information Science and Technology, Vol. 46 N. 6, pp 461-467.
Nisonger,
T.E. (1994), “A methodological issue concerning the use of social sciences
citation index journal citation reports impact factor data for journal
ranking”, Library Collections, Acquisition and Technical Services, Vol.
18 No. 4, pp. 447-458.
Nisonger,
T.E. (2000), “Use of the journal citation reports for serials management in
research libraries: An investigation of the effect of self-citation on journal
rankings in library and information science and genetics”, College and
Research Libraries, Vol. 61 No. 3, pp. 263-275.
Nisonger,
T.E. (2004), “The benefits and drawbacks of impact factor for journal
collection management in libraries”, Serials Librarian, Vol 47 No. 1-2,
pp. 57-75.
Oppenheim, C. (1996), “Do citations count? Citation indexing
and research assessment exercise”, Serials,
Vol. 9 No. 2, pp. 155-161.
Pendlebury,
D.A. (2009), “The use and misuse of journal metrics and other citation
indicators”, Archivum Immunologiae Et Therapiae Experimentalis, Vol. 57 No. 1, pp. 1-11.
Rousseau,
R. (2001), “Journal evaluation: Technical and practical issues”, Library
Trends, No. 50 No. 3, pp. 418-439.
Rousseau,
R. (2005), “Median and percentile impact factors.: A set of new indicators”, Scientometrics,
Vol. 63 No. 3, pp. 431-441.