Ó The Author 2007. Published by Oxford University Press. All rights reserved. For permissions, please e-mail: journals.permissions@oxfordjournals.org. doi:10.1093/fampra/cmm061 Measuring the impact of family medicine research: scientific citations or societal impact? Mieke L. van Driela, Manfred Maierb and Jan De Maeseneera van Driel ML, Maier M and De Maeseneer J. Measuring the impact of family medicine research: scientific citations or societal impact? Family Practice 2007; 24: 401–402. The ultimate goal of research in medicine, a spectrum starting with basic biomedical research via clinical trials to implementation research, is to improve the health and the quality of life of individuals and communities. Research should therefore be of the very best quality and assessment of its quality is important. Traditionally, research and researchers are evaluated by means of the ‘scientific impact’ of research output, i.e. publications in biomedical journals. The higher the impact factor of a journal, the better the research scores. Journals know how they can polish up their impact factor and for some this is even a deliberate policy.1,2 Increasingly, journal impact factors are also used to evaluate academics nominated for promotion and to allocate research funds. Impact factors are considered as an objective way to judge the quality of research/researchers. However, is the journal impact factor an appropriate and sufficient indicator of quality? There are reasons for doubt.3,4 The journal impact factor is calculated by dividing the total number of cited articles in the previous 2 years by the number of ‘citable’ articles in the journal during the same period. It is not clear what makes an article citable and the commercial company that creates and manages the journal impact factor, Thomson Scientific, is not very transparent in the criteria it uses to select journals for inclusion in the database either. The Journal Citation Reports database only covers a fraction of all the available biomedical journals; approximately 5000 compared with over 33 000 that are indexed in Medline.5,6 The highest impact factors can be found among the journals in the field of basic medical sciences such as molecular biology and biochemistry. In these highly dynamic research fields, a large proportion of citations are captured in the short term used to calculate the impact factor.7 In fields with a more durable literature, like epidemiology, public health and also family medicine, citations are usually spread over a longer period of time, resulting in lower impact factors. Moreover, the domain of family medicine is nearly invisible in the database. Only eight family medicine journals are listed in the Journal Citation Reports, all located at the tail of the 105 journals that make up the domain of general and internal medicine.8 In addition, there is a regional bias since in most domains North American journals have the highest impact factor. This is illustrated by a recent development in the family medicine journal landscape. With a first impact factor of 3.803, the Annals of Family Medicine (launched only in 2003) immediately surpasses all other family medicine journals.8 We congratulate the journal for this achievement that will be an important boost for family medicine research. However, it also shows that impact factors have no relation to the position of family medicine or research in the region, since family medicine (research) is more developed in some European countries than in the US. So there is a bias of discipline and region, but is it still a good indicator of research quality? Although some argue that a high impact factor is generally linked to better quality of the articles, this is not apparent in all domains.9–11 Usually, only a minority of articles account for the majority of citations.7,12 In fact, the journal impact factor is nothing more than an index of how often a specific journal has been cited in a selected group of journals. Some have even suggested changing its name to ‘citation rate index’13 or ‘journal citation ratio’,14 since the word ‘impact’ suggests more than what it actually stands for. The journal impact factor has an impact on how research and researchers are valued by their academic peers and on the allocation of research funds, but does it have an impact on what research is ultimately about, improving quality of life for all citizens? It has been suggested to pay attention to the ‘societal impact’ of research as well. Adding to the a Department of Family Medicine and Primary Health Care, Ghent University, Ghent, Belgium and bDepartment of General Practice and Family Medicine, Center for Public Health, Medical University of Vienna, Vienna, Austria. Correspondence to Mieke L van Driel, Department of Family Medicine and Primary Health Care, Ghent University, De Pintelaan 185, UZ 1K3, B-9000 Ghent, Belgium; Email: mieke.vandriel@ugent.be 401 402 Family Practice—an international journal assessment indicators like presenting research results to a non-scientific audience, communication through the media, participating in the development of practice guidelines or health policy or involvement in training health professionals could be a first step toward a more comprehensive approach of research quality.15 Based on these considerations, three categories for relevance of research output have been proposed.16 The first category refers to the local usually discipline-orientated relevance. If a publication influences national policy or has national relevance, it is classified as Category 2. The highest Category 3 covers relevance at an international level (e.g. European guidelines). Such an approach would not only acknowledge the importance of the societal relevance of research, but would also provide an incentive for researchers to improve their performance in this respect. A team of Flinders University in Australia shows how end users can be involved in evaluating if the research has been successfully applied to achieve social, economic, environmental and/or cultural outcomes.17 They studied the impact of four primary health care projects and found that the number of peer-reviewed publications (and their impact factor) was not indicative of the impact of the projects in society. In fact, the project that had led to important health policy reforms did not have any peer-reviewed publications. Family medicine research often deals with the implementation of research findings from basic science or other disciplines. Interventions often take years to produce tangible results. By nature, the scientific impact factor of family medicine research is low, but its potential to influence the quality of our lives and to contribute to equity is high. Woolf illustrated how a small improvement in coverage of beneficial treatments can outweigh the effect of similar size improvement of the outcome of those treatments.18 Family medicine is where science and society meet. The challenge is now to develop ways to value and draw attention to the importance of societal impact of science. And it is up to us to take up this challenge. View publication stats Declaration Funding: None. Ethical approval: Not applicable. Conflicts of interest: None declared. References 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 Brown H. How impact factors changed medical publishing—and science. BMJ 2007; 334: 561–564. PloS Medicine Editors. The impact factor game. PloS Med 2006; 3: e291. Smith R. Commentary: the power of the unrelenting impact factor—is it a force for good or harm? Int J Epidemiol 2006; 35: 1129–1130. Håkansson A. The impact factor—a dubious measure of scientific quality. Scand J Prim Health Care 2005; 23: 193–194. Testa J. The Thomson Scientific Journal Selection Process. http:// www.scientific.thomson.com/free/essays/selectionofmaterial/ journalselection/. Garfield E. The significant scientific literature appears in a small core of journals. Scientist 1996; 10 (17): 13. Seglen PO. Why the impact factor of journals should not be used for evaluating research. BMJ 1997; 314: 498–502. ISI Web of Knowledge. Journal Citation Reports. Philadelphia: Thomson Scientific, 2006. Gluud LL, Sørensen TIA, Gøtzsche PC, Gluud C. The journal impact factor as a predictor of trial quality and outcomes: cohort study of hepatobiliary randomised clinical trials. Am J Gastroenterol 2005; 100: 2431–2435. Lee KP, Schotland M, Bacchetti P, Bero LA. Association of journal quality indicators with methodological quality of clinical research articles. JAMA 2002; 287: 2805–2808. Opthof T. Sense and nonsense about the impact factor. Cardiovasc Res 1997; 33: 1–7. Not-so-deep impact. Nature 2005; 435: 1003–1004. Hecht F, Hecht BK, Sandberg AA. The journal ‘‘impact factor’’: a misnamed, misleading, misused measure. Cancer Genet Cytogenet 1998; 104: 77–81. Kurmis AP. Understanding the limitations of the journal impact factor. J Bone Joint Surg 2003; 85-A: 2449–2454. Smith R. Measuring the societal impact of research. BMJ 2001; 323: 528. Maier M. Men’s health and gender medicine: scientific impact or societal impact? J Mens Health Gend 2006; 3: 330–331. Kalucy L, McIntyre E, Jackson Bowers E. Primary Health Care Research Impact Project. Final Report Stage 1. Flinders University, http://www.phcris.org.au/phplib/filedownload. php?file=/elib/lib/downloaded_files/publications/pdfs/phcris_ pub_3338.pdf (accessed on 27 September 2007). Woolf SH, Johnson RE. The break-even point: when medical advances are less important that improving the fidelity with which they are delivered. Ann Fam Med 2005; 3: 545–552.