Comparative Analysis of Research Performance of Medical Universities Based on Qualitative and Quantitative Scientometric Indicators

ORIGINAL ARTICLE Comparative Analysis of Research Performance of Medical Universities Based on Qualitative and Quantitative Scientometric Indicators Zoleikha Ranjbar-Pirmousa1, Narges Borji-Zemeidani1, Mirsaeed Attarch2, Shadman Nemati1,2, Farzaneh Aminpour3 3 1 Deputy of Research and Technology, Guilan University of Medical Sciences, Rasht, Iran 2 Medical Education Research Center, Guilan University of Medical Sciences, Rasht, Iran Department of Research and Technology, Tehran University of Medical Sciences, Tehran, Iran Received: 30 Mar. 2019; Accepted: 28 Jun. 2019 Abstract- Evaluation of the research status of the academic institution provides it with the possibility of accurate research policymaking. Scientometric indicators are important tools for evaluating scientific activities of individuals, groups, and institutions. The current research aims to analysis the research performance of medical universities in Northern Iran based on quantitative and qualitative scientometric indicators. In this cross-sectional descriptive study, the Scopus-indexed scientific documents provided by medical universities in the Northern Iran have been studied in terms of number of publications, number of citations, average number of Citations per Publication (C/P), Field-Weighted Citation Impact (FWCI), scientific collaborations, the number of in top 10% citation percentile, and the number of publications in top 10% journal percentile according to CiteScore, Source Normalized Impact per Paper (SNIP), and SCImagoJournal Rank (SJR) indicators during a five years period. According to the findings, Mazandaran University of Medical Sciences has gained the highest position in terms of the number of publications and citations, number of publications with international and national collaborations, and academic-industrial collaborations., while Golestan University of Medical Sciences has gained a higher position in terms of scientific outputs in top 10% citation percentile and journal percentile, CiteScore, SNIP, SJR and C/P. In terms of the FWCI indicator, Golestan University of Medical Sciences has achieved the highest value. Considering academic status and research capabilities of medical universities in the Northern Iran, increasing academic-industrial collaboration, expanding academic collaboration with superior universities and institutions around the world can be effective in increasing the quality of research and upgrading academic ranks of universities at national, regional and international levels. © 2019Tehran University of Medical Sciences. All rights reserved. Acta MedIran 2019;57(7):448-455. Keywords: Research assessment; Science production; Information management; Scientometrics, Bibliometrics; Iran Introduction Nowadays, the position and global authority of the countries and their basis for comprehensive development depend on the production of knowledge and research-based achievements, in addition to their applications at the global level. Therefore, giving importance to research and increasing research activities in each and every country results in their development and progress; and, it provides them with self-sufficiency and real independence (1). On the other hand, research is considered as one of the main missions of the medical faculties (2). Specifying the status of scientific outputs and advancements in various subject domains may be indicative of a comprehensive picture of the type of scientific activity performed by researchers and authors in related fields; and, it may result in the identification of the weakness and strength points of various research cases performed (3). Scientometric studies can help research policymakers in allocating budget, creating a balance between budget and costs, appointing appointments, and promoting researchers and ranking academic institutes (4).Scientometrics indicators are important tools for evaluating scientific activities (5). These indicators are based on the following four variables: creators, scientific Corresponding Author: F. Aminpour Department of Research and Technology, Tehran University of Medical Sciences, Tehran, Iran Tel: +98 21 81633686, Fax: +98 21 81633698, E-mail address: f.aminpour@gmail.com Z. Ranjbar-Pirmousa, et al. outputs, references, and citations received by scientific works (6). Meanwhile, citation indicators play an important role in the evaluation of scientific activities (7). Citation analysis can provide clear information on the scientific activities of an individual, research group, journal, or higher education institution (8). In addition to the citation count, the number of most cited papers could also be used as an indicator for the review of research quality (9,10). Simultaneous use of quantitative indicators of scientific outputs and qualitative indicators of publications in the top 10% citation percentile in addition to the FWCI which measures the citation level of the scientific outputs of a particular country, university, or researcher are important factors for assessment of universities. Field-Weighted Citation Impact (FWCI) is a Snowball Metric which takes into account the differences in research behavior across disciplines. It also accounts for field-dependent citation differences and thus can be used across different disciplines. FWCI is defined as the ratio of the citations actually received by denominator's output, and the average citations received by all other similar publications. An FWCI=1 means the output performs just as expected for the global average. An FWCI> 1 means the output is cited less than expected by the global average (11). Currently, rating indicators of publications such as SJR (SCImago Journal Rank), SNIP (Source Normalized Impact per Paper), and Cite Score in Scopus database can be considered valid indicators for assessing the performance of universities in terms of publications. According to Glänzel et al., The compilation pattern of scientific publications and the tendency of researchers to contribute to research activities vary from one field to another (12) Scientometric indicators are frequently used to evaluate the researchers, disciplines, universities, countries and regions based on their scientific performance (13). Numerous studies are considered as evidence of increasing international scientific contributions of researchers in various fields of medical sciences (14,15). On the other hand, many studies have been done on the effect of scientific cooperation on scientific outputs' quality increase in various countries. Pečlin et al., believe that those papers, written through the international scientific contribution in the field of medical sciences, have more chance of being published in qualitative journals and turned to the most cited papers. These researchers consider international collaborations as the cause of increasing access to scientific outputs (16). A study conducted on scientific outputs in the field of medical sciences in Malaysia showed that papers generated by international collaborations received more citations on average compared to those produced by individuals or those derived from domestic cooperation (17). The results from another study on the international collaboration of Indian researchers in the field of medical sciences showed that these papers have been published in journals with higher quality and have received more citations (18). This way, it seems that academic cooperation can be applied as an important indicator in the university's and academic institutions' research evaluation. During the last decades, Iran has experienced remarkable development in various fields especially in medical science. Iranian universities have played a major role in this development by supporting research and knowledge dissemination (19). In 2017, the Iranian trends in health research outputs and their contribution to total science products during the period 2000-2014 were evaluated through a scientometric study. The study showed that 237,056 scientific documents had been published during the 15 years period, of which 81,867 (34.53%) were assigned to fields related to health (20). Eftekhari and colleagues ranked research production of Iranian medical universities based on international indicators. They reported that about 17% of papers from Iranian medical universities published in top-ranked journals and 15% published with international collaborations. The average paper per faculty member was 1.14, according to their study (21). In 2017, Aldieri and colleagues investigated the impact of internal and external research collaborations on the scientific performance to assess the performance of universities in some European countries, including Germany, France, Italy, the UK and Russia in the Scopus database. They considered the number of publications and the fieldweighted citation impact and the publication share in 10% of most cited articles to determine the extent to which the internal and external institutional cooperation's impact is sensitive to the geographical dimension of the data (22). Scientific outputs of the Mazandaran University of Medical Sciences in the Scopus database (1992-2013) have been mapped by Riahi et al., Their findings showed that scientific outputs of the university had had positive quantitative growth; however, a large number of papers has been published in journals with low impact factor (23). The results of another research showed that the Mazandaran University of Medical Sciences had a better status in terms of H-index and scientific outputs during 2005-2010, compared with other medical Acta Medica Iranica, Vol. 57, No. 7 (2019) 449 Comparative analysis of research performance of medical universities universities in Northern Iran (24). In terms of science production and the average number of citations per publications in the Web of Science citation database, this university positioned at the top of other universities in Northern Iran (25). On the other hand, numerous cases of research are indicative of the expansion of international scientific collaborations in Iran (26,27). The current study used a wide range of quantitative and qualitative scientometric indicators including number of publications, number of citations they received, the average citations per publication, FWCI indicator, institutional, national, and international levels of scientific collaboration, academic-industrial collaboration, publications in top 10% citation percentile, and publications in top 10% journal percentile according to CiteScore, SNIP, and SJR indicators. FWCI measures the relationship between the citation level of scientific outputs of a particular country, university, or researcher; and, in fact, it shows the ratio of received citations to the global average in a subject field, type of article, and year of publication. Publications in the top 10% citation percentile show those scientific outputs of an organization or country with a high percentage of citation received from among the voluminous scientific outputs. International collaborations also measure that group of scientific outputs produced by at least two authors and two countries' affiliation. This research analyses the research performance of Iranian medical universities in the Northen Iran in a way that the weakness and strength points of the research procedure would be identified in these universities. It can help research policymakers to plan for necessary actions and make appropriate interventions in terms of improving the scientific status of universities. Materials and Methods In this cross-sectional descriptive study, the scientific outputs of the medical universities in the Northern Iran including Guilan, Mazandaran, Babol and Golestan Universities of Medical Sciences in Scopus database have been studied for a five years period from the beginning of 2012 to end 2016 To extract data, the organizational affiliation of each of the aforementioned universities has first been searched separately in the "affiliation" field of the Scopus database. Next, each university's records during 2012-2016 have been studied separately, based on the year of publication, number of citations, and the average number of citations per publication. The SciVal citation analysis database has been applied for computing of FWCI, and academicindustrial collaborations. The results reported in the forms of graphs and tables. Results According to the results the Mazandaran University of Medical Sciences has had the highest number of scientific outputs (2364) and citations received (13386) during the five years period of the study. The lowest number of indexed papers (843) and the lowest level of received citations (3389) in Scopus have been related to Golestan and Guilan Universities of Medical Sciences, respectively. Highest (6.9) and lowest (3.2) average numbers of citations to scientific outputs have been related to Golestan and Babol Universities of Medical Sciences, respectively (Table 1). Table 1. Citation Analysis of the North Iranian Medical Universities Publications University Mazandaran University of Medical Sciences BabolUniversity of Medical Sciences GuilanUniversity of Medical Sciences GolestanUniversity of Medical Sciences 2012 2013 2014 2015 2016 2012-2016 P C C/P P C C/P P C C/P P C C/P P C C/P P C C/P 299 1882 6.3 459 2679 5.8 486 2627 5.4 500 4981 10 620 1217 2 2364 13386 5.7 217 1032 4.8 202 1008 5 241 539 3.5 249 623 2.5 315 373 1.2 1224 3875 3.2 135 837 6.2 159 823 5.2 164 777 4.7 218 636 2.9 248 316 1.3 924 3389 3.7 138 1103 8 142 829 5.8 154 680 4.4 171 643 3.8 238 1631 6.9 843 4886 5.8 Various types of collaborations in terms of scientific production of universities in Northern Iran are presented and compared in Table 2. The results showed that among the Iranian medical universities, the academic- 450 Acta Medica Iranica, Vol. 57, No. 7 (2019) industrial share of Golestan and Mazandaran Universities had been 11 publications. Meanwhile, Guilan and Babol Universities of Medical Sciences have produced no article in cooperation with industry. Z. Ranjbar-Pirmousa, et al. Mazandaran University of Medical Sciences has gained a higher position than the other universities respectively, with 220, 1198, and 917 publications, in terms of international, national, and institutional collaborations within the time interval of the study. This university also gained a higher position in terms of number of citations and the average number of citations per publication resulted from international and national collaboration, compared to other medical universities of Northern Iran. In terms of FWCI, Mazandaran University of Medical Sciences has gained the highest position among other universities with respectively 5.84, 0.68, and 92.77 publications due to international, national, and academic-industrial collaborations. In terms of documents with one author, Babol University of Medical Sciences has had the highest FWCI indicator (0.94), the highest number of citations (351), and the highest average citations per publication (16.7). Table 2. Academic Collaboration of the Northern Iranian Medical Universities University Mazandaran University of Medical Sciences Golestan University of Medical Sciences Guilan University of Medical Sciences Babol University of Medical Sciences Indicator International Collaboration National Collaboration Institutional Collaboration Single Authorship (No Collaboration) AcademicIndustrial Collaboration Publication 220 1198 917 29 11 Percentage 9.3 50.7 38.8 1.2 0.5 Citation Citation/Publication FWCI Publication 5802 26.4 5005 4.2 2480 2.7 99 3.4 4251 386.5 5.84 0.68 0.43 0.6 92.77 147 492 179 25 11 Percentage 17.4 58.4 21.2 3 1.3 Citation Citation/Publication FWCI Publication Percentage 2247 2013 560 66 1180 15.3 4.1 3.1 2.6 107.3 5.65 0.67 0.56 0.33 62.25 102 477 333 10 0 11 51.6 36.3 1.1 0 Citation 761 1668 905 55 0 Citation/Publication FWCI 7.5 3.5 2.7 5.5 0 1.2 0.57 0.44 0.46 0 Publication Percentage Citation Citation/Publication FWCI 83 6.8 563 46 557 45.5 21 1.7 0 0 410 1743 1371 351 0 4.9 0.76 3.1 0.48 2.5 0.41 16.7 0.94 0 0 Results provided in Table 3 show that Golestan University of Medical Sciences has had the highest number of publications in the top 10% journal percentile in terms of CiteScore, SNIP, and SJR indicators, respectively, with 9, 3.8 and 8.1%. The lowest number of publications in the top 10% journal in terms of the CiteScore indicator has been 3.1% and related to the Babol University of Medical Sciences. As far as publications in the top 10% citation percentile are concerned, the highest (190) and the lowest (53) numbers of publications produced by the Mazandaran and Guilan Universities of Medical Sciences respectively. The highest rate of FWCI (3.68) for publications of the universities understudy has been related to 2016 and to the Golestan University of Medical Sciences. Overall, the FWCI rate for the five-year period has been 1.5, 1.07, 0.59, and 0.47 for Golestan, Mazandaran, Guilan, and Babol Universities of Medical Sciences respectively (Table 4). Graph (1) shows the scientific outputs of the publications in the top 10% citation percentile of the medical universities in Northern Iran, separately for each year of the period under study. According to the results and in comparison with their previous year, Babol (63), Golestan (39), and Guilan (23) The Acta Medica Iranica, Vol. 57, No. 7 (2019) 451 Comparative analysis of research performance of medical universities universities have shown a growing trend, while, the Mazandaran University of Medical Sciences has shown a severe drop (43) interms of publications in the top 10% citation percentile in 2016 compared to that of 2015. Table 3. Comparison of Highly Cited Publications of the Northern Iranian Medical Universities University Mazandaran University of Medical Sciences Golestan University of Medical Sciences Babol University of Medical Sciences Guilan University of Medical Sciences Publications in Top 10% Journal Percentile (CiteScore) Number Percentage Publications in Top 10% Journal Percentile (SNIP) Number Percentage Publications in Top 10% Journal Percentile (SJR) Number Percentage Publications in Top 10% Citation Percentile Number Percentage 112 4.7 52 2.2 48 2.3 190 8.0 76 9.0 32 3.8 68 8.1 87 10.3 38 3.1 20 1.6 36 3.0 64 5.2 46 5.0 11 1.2 32 3.5 53 5.7 Table 4. Ranking Northern Iranian Medical Universities According to Field Weighted Citation Impact Rank 1 2 3 4 University GolestanUniversity of Medical Sciences Mazandaran University of Medical Sciences GuilanUniversity of Medical Sciences Babol University of Medical Sciences FWCI 2012 FWCI 2013 FWCI 2014 FWCI 2015 FWCI 2016 FWCI 2012-2016 0.63 0.58 0.64 0.74 3.68 1.50 0.47 0.68 0.77 2.29 0.88 1.07 0.47 0.55 0.65 0.61 0.63 0.59 0.33 0.45 0.46 0.56 0.53 0.47 Graph 1. The Trend of the Northern Iranian Medical Universities Publications in Top 10% Citation Percentile 452 Acta Medica Iranica, Vol. 57, No. 7 (2019) Z. Ranjbar-Pirmousa, et al. Discussion The results indicate that during the period 20122016, the Mazandaran University of Medical Sciences has had the highest rate of scientific output with the highest citation rate. According to Riahi and colleagues, the Mazandaran University of Medical Sciences showed an increase in scientific publications indexed by the Scopus database during the period 1992-2013 (23). Results from another study also showed that the Mazandaran University of Medical Sciences was positioned the highest rank of universities in Northern Iran, in terms of science productions and the average number of citations per document in Web of Science (25). According to the present study, the average number of citations per publication in Golestan University of Medical Sciences has been higher than other medical universities in Northern Iran. An increasing number of citations would indicate higher quality of scientific outputs (28,29); therefore, it seems that the quality of the Golestan University of Medical Science publications has been higher, in comparison to the other medical universities of Iran. In terms of scientific collaborations, Mazandaran and Golestan Universities of Medical Sciences have produced respectively the largest number of publications produced underinternational collaboration. According to the other researches, international research collaborations could improve the number of scientific papers (30,31). This way, it seems that creating a proper ground for the expansion of international collaborations may result in a quantitative and qualitative increase in the scientific output of Iranian medical universities. The present study showed that those publications from Mazandaran University of Medical Sciences produced by international and national as well as academic-industrial collaborations had been more desirable status in terms of citations, citations per publication and FWCI indicators. It seems that the number of citations received by each paper has a direct relationship with FWCI, in various types of academic collaborations. According to Bornamann, the citation impact of papers has a relationship with academic collaborations (32). Most cited papers are written by more researchers; and, they are mostly written through international collaborations (16-18,31,33). Scientific collaboration increases the quality of research conducted through international collaboration (34). Moreover, the FWCI would be increased through international academic collaborations (35). Therefore, it seems that scientific collaborations and FWCI can be considered as qualitative indicators in evaluating and ranking academic institutions and universities. According to CiteScore, SNIP, and SJR indicators, the Golestan University of Medical Sciences publications in the top 10% of the journal's percentile have been higher than other universities in the medical sciences in Northern Iran. Meanwhile, Golestan, Giulan and Babol universities of medical sciences have shown a growing trend of publications in the top 10% citation percentiles during 2012-2016, with an even steeper rising slope for Babol University during 2015-2016 (14 to 63 most citations), compared to other medical universities in Northern Iran. However, the growing trend of the Mazandaran University of Medical Sciences from 2012 to 2015 turned into a downward trend, in 2016. It seems that selecting top journals in each field by academic researchers may result in an increasing number of papers used and a number of citations they receive. On the other hand, it seems that there is a direct relationship between the FWCI of scientific outputs in a university and its publications in the top 10% journal percentile. This can be evidence of the effect of publications in top journals on scientific works being more viewed. More importantly, it can result in a growing number of citations and papers by the FWCI rate. In general, Guilan, Mazandaran, Babol and Golestan Universities of Medical Sciences need to elevate their international and academic-industrial collaborations in order to improve the quality of their scientific publications and their positions in academic ranking systems. An analytical study of effective and preventive factors in research collaborations of faculty members of Iranian medical universities can pave the way for improving academic collaborations at national, regional, and international levels. Novel research policies should be designed to empower the research abilities of faculty members as well as their information knowledge. Acknowledgments The funding from Guilan University of Medical Sciences to support this research project is gratefully acknowledged. (Grant No. 1361878825) References 1. 2. Adkins D, Budd J. Scholarly productivity of US LIS faculty. Libr Inf Sci Res 2006;28:374-89. Dakik HA, Kaidbay H, Sabra R. Resrarch Productivity of the Medical Faculty at The American Univeristy of Acta Medica Iranica, Vol. 57, No. 7 (2019) 453 Comparative analysis of research performance of medical universities 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. Beirut. Postgrod Med J 2006;82:462-4. Bookstein A. Scientometrics: New opportunities. Scientometrics 1994;30:455-60. Yazdani K, Nedjat S, Rahimi-Movaghar A, Ghalichee L, Khalili M. Scientometrics: Review of concepts, applications, and indicators. Iranian J Epidemiol. 2015;10:78-88. King DA. The scientific impact of nations. Nature 2004;430:311-6. Glänzel W. On the h-index-Amathematical approach to a new measure of publication activity and citation impact. Scientometrics. 2006;67:315-21. Garfield E. The history and meaning of the journal impact factor. Jama 2006;295:90-3. Nightingale J, Marshall G. Citation analysis as a measure of article quality, journal influence and individual researcher performance. Radiography 2012;18:60-7. Basu A. Using ISI's' Highly Cited Researchers' to obtain a country level indicator of citation excellence. Scientometric 2006;68:361-75. Levitt J, Thelwall M. Patterns of annual citation of highly cited articles and the prediction of their citation ranking: A comparison across subjects. Scientometrics 2008;77:41-60. Your metrics questions answered: Q&A from research impact metrics for librarians webinar. libraryconnect (Accessed May 24, 2016, athttps://libraryconnect.elsevier.com/articles/yourmetrics-questions-answered-qa-research-impact-metricslibrarians-webinar.) Glänzel W, De Lange C. Modelling and measuring multilateral co-authorship in international scientific collaboration. Part II. A comparative study on the extent and change of international scientific collaboration links. Scientometrics 1997;40:605-26. Aminpour F, Kabiri P, Boroumand MA, Keshtkar AA, Hejazi SS. Iranian Medical Universities in SCIE: evaluation of address variation. Scientometrics 2010; 85(1):53-63. Chimhundu C, de Jager K, Douglas T. Sectoral collaboration networks for cardiovascular medical device development in South Africa. Scientometrics 2015;105:1721-41. Hu J, Chen J, Karbwang J, Hirajama K. National and international collaboration in Chinese medical academic research. International j pharmaceut med 2006;20:373-8. Pečlin S, Južnič P, Sajko M, Stare J. Effects of international collaboration and status of journal on impact of papers. Scientometrics 2012;93:937-48. Low W, Ng K, Kabir M, Koh A, Sinnasamy J. Trend and impact of international collaboration in clinical medicine 454 Acta Medica Iranica, Vol. 57, No. 7 (2019) 18. 19. 20. 21. 22. 23. 24. 25. 26. 27. 28. 29. 30. 31. papers published in Malaysia. Scientometrics 2014;98:1521-33. Kaur H, Mahajan P. Collaboration in medical research: a case study of India. Scientometrics 2015;105:683-90. Aminpour F, Kabiri P, Heydari M. Academic contribution to the scientific productivity: a case study. J Res Med Sci 2009;14:393-5. Djalalinia S, Peykari N, Eftekhari MB, Sobhani Z, Laali R, Qorbani OA, et al. Contribution of health researches in national knowledge production: A scientometrics study on 15-year research products of Iran. Int J Prev Med 2017;8:27. Eftekhari MB, Sobhani Z, Eltemasi M, Ghalenoee E, Falahat K, Habibi E, et al. Research ranking of Iranian universities of medical sciences based on international indicators: An experience from I.R. of Iran. Arch Iran Med 2017;20:673-9. Aldieri L, Kotsemir M, Vinci C. The impact of research collaboration on academic performance: An empirical analysis for some European countries. Socioecon Plann Sci 2018;62:13-30. Riahi A, Siamian H, Zare A, Yaminfirooz M. Mapping the scientific productions of mazandaran university of medical sciences in Scopus database in 1992-2013. J Mazandaran Univ Med Sci 2015;24:395-400. Siamian H, Yamimi-Firooz M, Vahedi M, Aligolbandi K. Scientific Production of medical sciences universities in North of Iran. Acta Inform Med 2013;21:113-5. Jahani M, Yaminfirooz M. Self-citation of Medical and Non-medical Universities in Northern Iran. Acta Inform Med 2016;24:401-4. Hayati Z, Didegah F. International scientific collaboration among Iranian researchers during 1998-2007. Library Hi Tech 2010;28:433-46. Moed H. Iran’s scientific dominance and the emergence of South-East Asian countries as scientific collaborators in the Persian Gulf Region. Scientometrics 2016;108:30514. Mazlish B. The quality of the quality of science: An evaluation. Sci Technol Human Values 1982;7:42-52. Pak C, Yu G, Wang W. A study on the citation situation within the citing paper: citation distribution of references according to mention frequency. Scientometrics 2018;114:905-18. Sooryamoorthy R. Collaboration and publication: How collaborative are scientists in South Africa? Scientometrics 2009;80:419-39. De la Flor-Martínez M, Galindo-Moreno P, SánchezFernández E, Abadal E, Cobo M, Herrera-Viedma E. Evaluation of scientific output in Dentistry in Spanish Universities. Medicina oral,patologia oral y cirugia bucal Z. Ranjbar-Pirmousa, et al. 2017;22:491-9. 32. Bornmann L. Is collaboration among scientists related to the citationimpact of papers because their quality increases with collaboration? Ananalysis based on data from F1000Prime and normalized citation scores. Journal of the Association for Information Science and Technology. 2017;68:1036-47. 33. Aksnes D. Characteristics of highly cited papers. Researchevaluation. Res Eval 2003;12:159-70. 34. Leydesdorff L, Wagner C. International collaboration in science and the formation of a core group. J Informetr 2008;2:317-25. 35. Khor KA YL. Influence of internationalco-authorship on the research citation impact of young universities. Scientometrics 2016;107:1095-110. Acta Medica Iranica, Vol. 57, No. 7 (2019) 455