The FASEB Journal • Life Sciences Forum Comparison of SCImago journal rank indicator with journal impact factor Matthew E. Falagas,*,†,‡,1 Vasilios D. Kouranos,* Ricardo Arencibia-Jorge,§ and Drosos E. Karageorgopoulos* *Alfa Institute of Biomedical Sciences (AIBS), Athens, Greece; †Department of Medicine, Tufts University School of Medicine, Boston, Massachusetts, USA; ‡Department of Medicine, Henry Dunant Hospital, Athens, Greece; and §Network of Scientometrics Studies for Higher Education, National Scientific Research Center, Havana, Cuba The application of currently available sophisticated algorithms of citation analysis allows for the incorporation of the “quality” of citations in the evaluation of scientific journals. We sought to compare the newly introduced SCImago journal rank (SJR) indicator with the journal impact factor (IF). We retrieved relevant information from the official Web sites hosting the above indices and their source databases. The SJR indicator is an open-access resource, while the journal IF requires paid subscription. The SJR indicator (based on Scopus data) lists considerably more journal titles published in a wider variety of countries and languages, than the journal IF (based on Web of Science data). Both indices divide citations to a journal by articles of the journal, during a specific time period. However, contrary to the journal IF, the SJR indicator attributes different weight to citations depending on the “prestige” of the citing journal without the influence of journal self-citations; prestige is estimated with the application of the PageRank algorithm in the network of journals. In addition, the SJR indicator includes the total number of documents of a journal in the denominator of the relevant calculation, whereas the journal IF includes only “citable” articles (mainly original articles and reviews). A 3-yr period is analyzed in both indices but with the use of different approaches. Regarding the top 100 journals in the 2006 journal IF ranking order, the median absolute change in their ranking position with the use of the SJR indicator is 32 (1st quartile: 12; 3rd quartile: 75). Although further validation is warranted, the novel SJR indicator poses as a serious alternative to the well-established journal IF, mainly due to its openaccess nature, larger source database, and assessment of the quality of citations.—Falagas, M. E., Kouranos, V. D., Arencibia-Jorge, R., Karageorgopoulos, D. E. Comparison of SCImago journal rank indicator with journal impact factor. FASEB J. 22, 2623–2628 (2008) ABSTRACT The evaluation of the quality of research is important for various professional societies, individual scientists, scholarly institutions, and funding organizations (1). The quality of a scientific contribution is primarily estimated from the long-term impact that it has in science. The latter can be inferred from the citations in scientific articles that a contribution receives. These principles have been applied in the evaluation of scientific journals (2). The journal impact factor (IF), first conceived in 1955 by Eugene Garfield, the founder of the Institute for Scientific Information (ISI), has been extensively used in the past decades as an index of quality of scientific journals (3) and is based on citation analysis (4). Although the journal impact factor has been widely regarded as the best instrument for the evaluation of the quality of scientific journals, it has not been spared from criticism(5– 8). Main points of consideration regarding methodological aspects in the calculation of this index include the lack of assessment of the quality of citations (9), the inclusion of self-citations (10 –12), the poor comparability between different scientific fields (13), and the analysis of mainly English-language publications (14 –16). In fact, many researchers have proposed different approaches in the evaluation of the quality of scientific journals. The common point in most of these approaches is the assessment of the quality of citations received by a journal (17–20). The quality of citations can be estimated analyzing the networks of scientific papers with sophisticated mathematical algorithms (21). The PageRank algorithm, used in the evaluation of webpages by the popular Google search engines, has been proposed as an appropriate model for the evaluation of the quality of citations in scientific journals (22, 23). In fact, a group of researchers at the University of Washington developed a similar algorithm for the evaluation of the 1 Key Words: bibliometric analysis 䡠 quality of publications 䡠 bibliographic databases 䡠 mathematical computing 䡠 scientometrics 0892-6638/08/0022-2623 © FASEB Correspondence: Alfa Institute of Biomedical Sciences (AIBS), 9 Neapoleos St., 151 23 Marousi, Greece. E-mail: m.falagas@aibs.gr doi: 10.1096/fj.08-107938 2623 influence of scientific journals included in the Thompson Scientific Journal Citation Reports (JCR) dataset (24). Furthermore, another research group from Spanish Universities developed an indicator, named SCImago Journal Rank (SJR) indicator, for the assessment of the quality of scientific journals, applying the PageRank algorithm on the Scopus database (25). In this evolving context, we sought to identify and evaluate the main characteristics and differences between the widely used journal IF by ISI and the newly introduced SJR indicator. MATERIALS AND METHODS We searched (January 2008) in the official relevant Web sites for information regarding the main characteristics of the journal IF, provided by JCR through ISI Web of Science, Thomson Scientific, and the SJR indicator, provided by the SCImago journal and country rank Web site, and developed by the SCImago research group. We also searched for information pertinent to the characteristics of these two indices of quality of scientific journals in the official Web sites hosting the databases used by each one of the indices (ISI Web of Science and Scopus for journal IF and SJR indicator, respectively). In addition, we listed the journals with the top 100 journal IFs and retrieved information regarding their ranking in the SJR indicator list by matching their interna- tional standard serial number (ISSN). We also listed the journals with the top 100 SJR indicators and found their ranking in the list of journal IFs. Finally, we calculated the median value as well as the first and third quartile values of the absolute change in ranking order of the journals in the two top 100 lists (with the use of each one of the compared indices vs. the other). RESULTS The main characteristics of the evaluation of scientific journals in JCR and in SCImago journal rank are summarized in Table 1. The journal IF of a specific journal for a specific calendar year is defined by the total number of citations (references) by articles published during the specific year in “source” journals (potentially including the respective journal) to any article of the specific journal that was published during the preceding 2 yr, divided by the total number of “citable” articles published in the respective journal during the preceding 2 yr. Articles regarded as citable are mainly original research and review articles. The SJR indicator of a specific journal for a threecalendar-year period is calculated through an iteration process that computes the “prestige” gained by the journal through the transfer of prestige from all the TABLE 1. Main characteristics of the evaluation of scientific journals by journal citation reports and SCImago journal and country rank. Characteristic Journal citation reports Organization Source database Number of journals Languages of publication of journals Countries of publication of journals Countries of research origin Update Main indicator of quality of journals Reference period Citation window Journals providing citations Weight of citations Journal self-citations Articles considered to receive citations Subject classification Access Secondary indices, utilities a 2624 Imago journal and country rank Thomson Scientific Institute for Scientific Information (ISI) Web of Science ⬍7934a 30 SCImago research group Scopus (Elsevier B.V.) 71 97 Not available Weekly Journal impact factor (IF) 229 Daily SCImago journal rank (SJR) indicator 1 calendar year 2 preceding years “Source” journals (“cited-only” journals excluded) Equal 3 calendar years 3 past years All other journals Included “Citable” (research and review articles) 2 Editions (science and social sciences), 224 subject categories Restricted (paid subscription required after 1 month free use) Journal immediacy index, journal cited half-life, unified impact factor, 5-yr impact factor, selfcites, graphical representations 13,208 50 Depending on the “prestige” of the citing journal Not included All types 27 subject areas, 295 subject categories Open H Index, self-cites, country indicators, graphical representations 6166 science journals, 1768 social science journals. Vol. 22 August 2008 The FASEB Journal FALAGAS ET AL. other journals included in the network of journals, by their citations during the past 3 yr to all articles of the specific journal published in the past 3 yr, divided by the total number of articles of the specific journal during the 3 yr period in regard. The amount of prestige of each journal transferred to another journal in the network is computed by considering the percentage of citations of the former journal that are directed to articles of the latter journal (26). Table 2 presents the first 20 ranked journals of all categories with each one of the compared indices and their corresponding rank using the other index. Of the 20 journals with the highest journal IFs, 13 retain a position in the top 20 journals with the use of the SJR indicator, and vice versa. Regarding the top 100 journals in the 2006 journal IF ranking order, the median absolute change in their ranking position with the use of the SJR indicator is 32 (1st quartile: 12; 3rd quartile: 75). Conversely, regarding the top 100 journals with the current SJR indicator, the median absolute change in their ranking position with the use of the journal IF is 29 (1st quartile: 10.5; 3rd quartile: 65.5). The journals with the greatest relative increase in their ranking order in the SJR indicator compared to the journal IF were the journals Immunity, Molecular Cell, Annual Review of Cell and Developmental Biology, Cell, and Current Opinion in Cell Biology. The journals with the greatest relative stability in their ranking order from the journal IF to the SJR classification were Annual Review of Physiology, Annual Review of Biophysics and Biomolecular Structure, PLoS Biology, An- TABLE 2. Comparative rankings of the top 20 journals by journal impact factor and SCImago journal rank indicator SCImago journal rank indicator Journal impact factor Rank Value Journal title Value Rank 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 63,342 51,296 47,237 36,525 33,508 31,583 31,441 31,354 30,028 29,194 28,697 28,588 28,533 27,596 26,681 26,576 26,054 25,800 24,370 24,176 Ca-A Cancer Journal of Clinicians New England Journal of Medicine Annual Review of Immunology Annual Review of Biochemistry Reviews of Modern Physics Nature Reviews Cancer Physiological Reviews Nature Reviews Molecular Cell Biology Science Cell Nature Reviews Immunology Nature Medicine Annual Review of Neuroscience Nature Immunology Nature Annual Review of Cell and Developmental Biology Chemical Reviews Lancet Briefings in Bioinformatics Nature Genetics 7,275 3,649 22,439 16,100 2,689 9,159 7,866 12,240 5,338 15,224 11,101 7,226 8,678 12,484 6,203 14,193 2,245 1,652 2,535 9,083 19 51 1 2 79 9 16 6 30 3 7 20 11 5 23 4 93 134 84 10 3 4 10 16 14 8 11 33 6 20 13 31 21 56 45 7 53 32 1 12 47,237 36,525 29,194 26,576 27,596 31,354 28,697 18,306 31,583 24,176 28,533 19,098 24,077 14,033 15,050 31,441 14,299 18,485 63,342 28,588 Annual Review of Immunology Annual Review of Biochemistry Cell Annual Review of Cell and Developmental Biology Nature Immunology Nature Reviews Molecular Cell Biology Nature Reviews Immunology Immunity Nature Reviews Cancer Nature Genetics Annual Review of Neuroscience Annual Review of Genetics Cancer Cell Molecular Cell Genes and Development Physiological Reviews Current Opinion in Cell Biology Nature Cell Biology Ca-A Cancer Journal of Clinicians Nature Medicine 22,439 16,100 15,224 14,193 12,484 12,240 11,101 9,337 9,159 9,083 8,678 8,583 8,214 8,185 8,086 7,866 7,399 7,367 7,275 7,226 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 SCIMAGO JOURNAL RANK VS. JOURNAL IMPACT FACTOR 2625 nual Review of Plant Biology, and Blood. The journals that exhibited the greatest relative decrease in their ranking order in the SJR indicator compared to the journal IF were Annual Review of Astronomy and Astrophysics, JAMA–Journal of the American Medical Association, Nature Physics, Advances in Catalysis, and Behavioral and Brain Sciences. It should be mentioned that all five of the latter journals exhibited a fall of more than 4000 positions in their ranking order with the use of the SJR indicator compared to the journal IF. Exploring the considerable discrepancy in the rankings of the latter five journals, we noted as potential causes a computational error, the presence of another journal with the same title but different ISSN, lack of source data, and a divergence between the number of citable articles and of total articles or documents. Furthermore, one of the top 100 journals according to the journal IF classification (Trends in Ecology & Evolution) was not included in the SJR database. DISCUSSION The main differences between the journal IF and the SJR indicator derive mainly from differences in the scientific databases used as the sources of citations, as well as from differences in the methodology of estimation of these indices. The latter primarily regard the weight attributed to citations, the way of handling self-citations, the temporal window analyzed, and the type and number of the articles of a journal considered in the denominator of calculation of the aforementioned indices. In terms of utility, the main novelty introduced by the SJR indicator is open access. Regarding the differences in the scientific databases, on which the compared indices of evaluation of scientific journals are applied, Scopus includes a substantially larger collection of journals, originating from remarkably more countries and published in a greater variety of languages (27). Thus, in this regard it can be assumed that SJR may provide a more comprehensive estimation of the scientific value of journals, particularly so for those published in nonEnglish languages (16). This is why the latter category of journals receives a great percentage of the overall incoming citations from non-English journals (14, 15). The latter are rather underrepresented in the Web of Science database. Furthermore, the Web of Science takes into consideration citations originating from a subset of source journals (28), potentially excluding some journals published in non-English languages, a factor that may influence the evaluation of this category of journals (14). Yet, it should be mentioned that although Scopus includes a larger collection of non-English journals, the latter are still underrepresented, constituting ⬃15% of the total number of included journals (29). Moreover, the 2626 Vol. 22 August 2008 apparent advantage of Scopus in citation analysis originating from the breadth of its database is limited to the time period after 1996 for which citation analysis is available (30). Regarding the methodology of the calculation of the two compared indices, the most significant difference lies in the fact that the SJR indicator takes into account not only the absolute number but also the “quality” of citations received by a journal, whereas the journal IF considers incoming citations only in a quantitative manner. It is plausible that the articles of a journal have a greater impact in science if they are cited by journals of higher scientific quality. Such an analysis may not have been feasible in the past, though in today’s electronic era, powerful computational systems provide the opportunity to apply sophisticated algorithms for the evaluation of interactions between journals in a huge network or universe of scientific publications. However, the simple and comprehensible methodology used in the calculation of the journal IF, despite the controversy raised over the years for the output data (31, 32), has been regarded as one of the most favorable attributes of this long-used standard of reference in the field of analysis of scientific citations. It should be noted though that the simple methodology used in the calculation of the journal IF has allowed editors to use various practices aiming to increase the impact factor of their journals (5, 33, 34). The principal ones are probably the promotion of self-citations (10 –12, 35), the predilection for review articles (36), and the decrease in the total number of included articles (37). Notably, the use of the SJR indicator allows for the estimation of a journal’s impact without the influence of self-citations, since prestige can be transferred to a journal by all other journals but not by itself. Instead, in JCR providing the journal IFs, self-citation analysis for each journal can be separately performed. However, this factor is not incorporated in the calculation of the journal IFs. Regarding the weight of the different types of articles in the process of calculating the two compared indices of scientific journals, no provision has been made for differentiating between the value assigned to original research articles compared to review articles in any of the indices. One of the major shortcomings of the SJR indicator may be the fact that it divides the prestige gained by a journal, through the citations of its articles, to the total number of articles included, rather than to the number of citable articles, as is used in the calculation of the journal IF. Although the strategy used in the calculation of the SJR indicator may be mathematically valid, since in theory any published article can be cited and all citations are taken into account in the numerator of the fraction, in practice, article types such as correspondence articles, letters to the editor, commentaries, perspectives, news, obituaries, editorials, interviews, and tributes are unlikely to receive a significant number of citations (3). In The FASEB Journal FALAGAS ET AL. this regard, the scientific quality of journals that contain a large number of the latter types of articles, which may otherwise be of interest to the reader, are expected to be appreciably underestimated with the SJR indicator. Yet, it should be mentioned that neither is the journal IF an optimal index of quality of scientific journals regarding this issue, since it does not adjust for the fact that a great number of total citations addressed to a journal are received by a relatively short portion of the included articles (3, 38, 39). The effect of the differences in the time periods used for the assessment of the two herein compared indices of quality of scientific journals on the accuracy of the estimates cannot be directly inferred in the context of the present study. It should be noted though that both of the indices refer to the past three calendar-year periods, although through a different approach. This temporal window may be relatively short, particularly for journals with an appreciable interval between receipt of an article and publication (3, 40). An indisputable advantage of journal IF over any new index of evaluation of quality of scientific journals is tradition. On the one hand, authors compete for publishing the products of their research in highly ranked journals (41). On the other hand, journal editors elect to publish scientific articles that are expected to be highly cited (37). Thus, it can be assumed that journal IF rankings would have influenced accordingly the quality of the journals over the years in which they have been considered as the sole standard of reference (3). Although our study did not aim to systematically assess the comparability in journal rankings between the novel SJR indicator and the well-established journal IF index, we observed that the introduction of the SJR indicator does not bring about radical changes in this regard. In fact, half of the journals in the top 100 journal IF list are placed within a reasonable range of 32 ranking places in the SJR indicator journal list. Some striking inconsistencies observed in the rankings of the new SJR indicator, compared to the journal impact factor, should be attributed to the “precocity” of the new indicator rather than to systematic differences in the methodologies used. In fact, the two methodologies have been systematically compared in previous studies. A weighted PageRank algorithm applied in the same dataset used for the derivation of the journal impact factor provided results that correlated significantly with the later index (23). However, this correlation was moderately strong and differed between various scientific fields, for instance, for medical journals (9, 23). The SCImago journal rank indicator is a novel instrument for the evaluation of scientific journals that may challenge the established premiership of the journal IF in ranking scientific journals. It provides unrestricted (open) access, is based on a larger source journal database, and focuses on the quality of SCIMAGO JOURNAL RANK VS. JOURNAL IMPACT FACTOR citations that a journal receives by other journals, rather than the absolute number. However, the sophisticated methodology used in the calculation of the SJR indicator needs to be adequately validated, and certain characteristics may need to be reconsidered before definitive conclusions for its applicability could be drawn. It appears, though, that the election of one index or the other would be mostly a matter of whether the popularity or the quality of a journal is considered as the primary criterion for the evaluation of scientific journals. The authors have no conflicts of interest to declare. No funding was received for this study. REFERENCES 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. Smith, R. (2006) Commentary: the power of the unrelenting impact factor–is it a force for good or harm? Int. J. Epidemiol. 35, 1129 –1130 Gross, P. L., and Gross, E. M. (1927) College libraries and chemical education. Science 66, 385–389 Garfield, E. (2006) The history and meaning of the journal impact factor. JAMA 295, 90 –93 Garfield, E. (1972) Citation analysis as a tool in journal evaluation. Science 178, 471– 479 Garfield, E. (1996) How can impact factors be improved? BMJ 313, 411– 413 Hoeffel, C. (1998) Journal impact factors. Allergy 53, 1225 Cameron, B. D. (2005) Trends in the usage of ISI bibliometric data. Portal Libraries Acad. 5, 105–125 Falagas, M. E., Zouglakis, G. M., and Papastamataki, P. A. (2006) Trends in the impact factor of scientific journals. Mayo. Clin. Proc. 81, 1401–1402 Dellavalle, R. P., Schilling, L. M., Rodriguez, M. A., Van de, S. H., and Bollen, J. (2007) Refining dermatology journal impact factors using PageRank. J. Am. Acad. Dermatol. 57, 116 – 119 Miguel, A., and Marti-Bonmati, L. (2002) Self-citation: comparison between Radiologia, European radiology and radiology for 1997–1998. Eur. Radiol. 12, 248 –252 Fassoulaki, A., Papilas, K., Paraskeva, A., and Patris, K. (2002) Impact factor bias and proposed adjustments for its determination. Acta Anaesthesiol. Scand. 46, 902–905 Falagas, M. E., and Kavvadia, P. (2006) “Eigenlob”: self-citation in biomedical journals. FASEB J. 20, 1039 –1042 Postma, E. (2007) Inflated impact factors? The true impact of evolutionary papers in non-evolutionary journals. PLoS ONE 2, e999 Winkmann, G., Schlutius, S., and Schweim, H. G. (2002) Citation rates of medical German-language journals in Englishlanguage papers– do they correlate with the impact factor, and who cites? Klin. Monatsbl. Augenheilkd. 219, 72–78 Aleixandre-Benavent, R., Valderrama Zurian, J. C., AlonsoArroyo, A., Miguel-Dasit, A., Gonzalez de Dios, J., and de Granda Orive, J. (2007) [Spanish versus English as a language of publication and impact factor of Neurologia]. Neurologia 22, 19 –26 Mueller, P. S., Murali, N. S., Cha, S. S., Erwin, P. F., and Ghosh, A. K. (2006) The association between impact factors and language of general internal medicine journals. Swiss. Med. Wkly. 136, 441– 443 Kodrzycki, Y. K., and Yu, P. D. (2005) New Approaches to Ranking Economics Journals. Federal Reserve Bank of Boston Working Paper, 5–12 Liebowitz, S. J., and Palmer, J. P. (1984) Assessing the relative impacts of economics journals. J. Econ. Lit. 22, 77– 88 Palacios-Huerta, I., and Volij, O. (2004) The measurement of intellectual influence. Econometrica 72, 963–977 Pinski, G., and Narin, F. (1976) Citation influence for journal aggregates of scientific publications: theory, with application to the literature of physics. Inform. Process. Manage. 12, 297–312 2627 21. 22. 23. 24. 25. 26. 27. 28. 29. 30. 2628 Price, D. J. (1965) Networks of scientific papers. Science 149, 510 –515 Page, L., Brin, S., Motwani, R., and Winograd, T. (1998) The PageRank citation ranking: Bringing order to the web. Technical report, Stanford Digital Library Technologies Project SIDLWP-1999-0120 Bollen, J., Rodriquez, M. A., and Van de Sompel, H. (2006) Journal status. Scientometrics 69, 669 – 687 Bergstrom, C. (2007) Scholarly communication eigenfactor: measuring the value and prestige of scholarly journals. C&RL News 68, 5 SCImago (2007) SJR–SCImago Journal & Country Rank. Retrieved January 20, 2008, from http://www.scimagojr.com SCImago Research Group. Description of SCImago Journal Rank Indicator. Retrieved January 20, 2008, from http://www. scimagojr.com/SCImagoJournalRank.pdf Falagas, M. E., Pitsouni, E. I., Malietzis, G. A., and Pappas, G. (2008) Comparison of PubMed, Scopus, Web of Science, and Google Scholar: strengths and weaknesses. FASEB J. 22, 338 –342 Gallagher, E. J., and Barnaby, D. P. (1998) Evidence of methodologic bias in the derivation of the Science Citation Index impact factor. Ann. Emerg. Med. 31, 83– 86 De Moya-Aneg␱n, F., Chinchilla-Rodriguez, Z., Vargas-Quesada, B., Corera-Alvarez, E., Munoz-Fernandez, F. J., Gonzalez-Molina, A., and Herrero-Solana, V. (2007) Coverage analysis of Scopus: a journal metric approach. Scientometrics 73, 53–78 Bakkalbasi, N., Bauer, K., Glover, J., and Wang, L. (2006) Three options for citation tracking: Google Scholar, Scopus, and Web of Science. Biomed. Digit. Libr. 3, 7 Vol. 22 August 2008 31. 32. 33. 34. 35. 36. 37. 38. 39. 40. 41. Rossner, M., Van, E. H., and Hill, E. (2007) Show me the data. J. Cell Biol. 179, 1091–1092 Joseph, K. S., and Hoey, J. (1999) CMAJ’s impact factor: room for recalculation. CMAJ 161, 977–978 Hemmingsson, A., Mygind, T., Skjennald, A., and Edgren, J. (2002) Manipulation of impact factors by editors of scientific journals. Am. J. Roentgenol. 178, 767 The PLoS Medicine Editors (2006) The impact factor game. It is time to find a better way to assess the scientific literature. PLoS Med. 3, e291 Falagas, M. E., and Alexiou, V. G. (2007) Editors may inappropriately influence authors’ decisions regarding selection of references in scientific articles. Int. J. Impot. Res. 19, 443– 445 Andersen, J., Belmont, J., and Cho, C. T. (2006) Journal impact factor in the era of expanding literature. J. Microbiol. Immunol. Infect. 39, 436 – 443 Chew, M., Villanueva, E. V., and Van Der Weyden, M. B. (2007) Life and times of the impact factor: retrospective analysis of trends for seven medical journals (1994 –2005) and their editors’ views. J. R. Soc. Med. 100, 142–150 Weale, A. R., Bailey, M., and Lear, P. A. (2004) The level of non-citation of articles within a journal as a measure of quality: a comparison to the impact factor. BMC. Med. Res. Methodol. 4, 14 No authors listed (2005) Not-so-deep impact. Nature 435, 1003– 1004 Yu, G., Wang, X. H., and Yu, D. R. (2005) The influence of publication delays on impact factors. Scientometrics 64, 235–246 Saha, S., Saint, S., and Christakis, D. A. (2003) Impact factor: a valid measure of journal quality? J. Med. Libr. Assoc. 91, 42– 46 The FASEB Journal FALAGAS ET AL.