Ó The Author 2007. Published by Oxford University Press. All rights reserved. For permissions, please e-mail: journals.permissions@oxfordjournals.org.
doi:10.1093/fampra/cmm061
Measuring the impact of family medicine research:
scientific citations or societal impact?
Mieke L. van Driela, Manfred Maierb and Jan De Maeseneera
van Driel ML, Maier M and De Maeseneer J. Measuring the impact of family medicine research:
scientific citations or societal impact? Family Practice 2007; 24: 401–402.
The ultimate goal of research in medicine, a spectrum
starting with basic biomedical research via clinical trials to implementation research, is to improve the
health and the quality of life of individuals and communities. Research should therefore be of the very
best quality and assessment of its quality is important.
Traditionally, research and researchers are evaluated
by means of the ‘scientific impact’ of research output,
i.e. publications in biomedical journals. The higher
the impact factor of a journal, the better the research
scores. Journals know how they can polish up their impact factor and for some this is even a deliberate policy.1,2 Increasingly, journal impact factors are also
used to evaluate academics nominated for promotion
and to allocate research funds. Impact factors are considered as an objective way to judge the quality of research/researchers. However, is the journal impact
factor an appropriate and sufficient indicator of quality? There are reasons for doubt.3,4
The journal impact factor is calculated by dividing
the total number of cited articles in the previous 2
years by the number of ‘citable’ articles in the journal
during the same period. It is not clear what makes an
article citable and the commercial company that creates and manages the journal impact factor, Thomson
Scientific, is not very transparent in the criteria it uses
to select journals for inclusion in the database either.
The Journal Citation Reports database only covers
a fraction of all the available biomedical journals; approximately 5000 compared with over 33 000 that are
indexed in Medline.5,6 The highest impact factors can
be found among the journals in the field of basic medical sciences such as molecular biology and biochemistry. In these highly dynamic research fields, a large
proportion of citations are captured in the short term
used to calculate the impact factor.7 In fields with
a more durable literature, like epidemiology, public
health and also family medicine, citations are usually
spread over a longer period of time, resulting in lower
impact factors. Moreover, the domain of family medicine is nearly invisible in the database. Only eight family medicine journals are listed in the Journal Citation
Reports, all located at the tail of the 105 journals that
make up the domain of general and internal medicine.8 In addition, there is a regional bias since in most
domains North American journals have the highest
impact factor. This is illustrated by a recent development in the family medicine journal landscape. With
a first impact factor of 3.803, the Annals of Family
Medicine (launched only in 2003) immediately surpasses all other family medicine journals.8 We congratulate the journal for this achievement that will be
an important boost for family medicine research.
However, it also shows that impact factors have no relation to the position of family medicine or research
in the region, since family medicine (research) is more
developed in some European countries than in the
US. So there is a bias of discipline and region, but is it
still a good indicator of research quality? Although
some argue that a high impact factor is generally
linked to better quality of the articles, this is not apparent in all domains.9–11 Usually, only a minority of
articles account for the majority of citations.7,12 In
fact, the journal impact factor is nothing more than an
index of how often a specific journal has been cited in
a selected group of journals. Some have even suggested changing its name to ‘citation rate index’13 or
‘journal citation ratio’,14 since the word ‘impact’ suggests more than what it actually stands for. The journal impact factor has an impact on how research and
researchers are valued by their academic peers and on
the allocation of research funds, but does it have an
impact on what research is ultimately about, improving quality of life for all citizens?
It has been suggested to pay attention to the ‘societal impact’ of research as well. Adding to the
a
Department of Family Medicine and Primary Health Care, Ghent University, Ghent, Belgium and bDepartment of General Practice and Family Medicine, Center for Public Health, Medical University of Vienna, Vienna, Austria. Correspondence to Mieke L
van Driel, Department of Family Medicine and Primary Health Care, Ghent University, De Pintelaan 185, UZ 1K3, B-9000 Ghent,
Belgium; Email: mieke.vandriel@ugent.be
401
402
Family Practice—an international journal
assessment indicators like presenting research results
to a non-scientific audience, communication through
the media, participating in the development of practice guidelines or health policy or involvement in
training health professionals could be a first step toward a more comprehensive approach of research
quality.15 Based on these considerations, three categories for relevance of research output have been proposed.16 The first category refers to the local usually
discipline-orientated relevance. If a publication influences national policy or has national relevance, it
is classified as Category 2. The highest Category 3 covers relevance at an international level (e.g. European
guidelines). Such an approach would not only acknowledge the importance of the societal relevance
of research, but would also provide an incentive for
researchers to improve their performance in this respect. A team of Flinders University in Australia
shows how end users can be involved in evaluating if
the research has been successfully applied to achieve
social, economic, environmental and/or cultural outcomes.17 They studied the impact of four primary
health care projects and found that the number of
peer-reviewed publications (and their impact factor)
was not indicative of the impact of the projects in society. In fact, the project that had led to important
health policy reforms did not have any peer-reviewed
publications.
Family medicine research often deals with the implementation of research findings from basic science
or other disciplines. Interventions often take years
to produce tangible results. By nature, the scientific
impact factor of family medicine research is low,
but its potential to influence the quality of our lives
and to contribute to equity is high. Woolf illustrated
how a small improvement in coverage of beneficial
treatments can outweigh the effect of similar size
improvement of the outcome of those treatments.18
Family medicine is where science and society meet.
The challenge is now to develop ways to value and
draw attention to the importance of societal
impact of science. And it is up to us to take up this
challenge.
View publication stats
Declaration
Funding: None.
Ethical approval: Not applicable.
Conflicts of interest: None declared.
References
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
Brown H. How impact factors changed medical publishing—and
science. BMJ 2007; 334: 561–564.
PloS Medicine Editors. The impact factor game. PloS Med 2006; 3:
e291.
Smith R. Commentary: the power of the unrelenting impact factor—is
it a force for good or harm? Int J Epidemiol 2006; 35: 1129–1130.
Håkansson A. The impact factor—a dubious measure of scientific
quality. Scand J Prim Health Care 2005; 23: 193–194.
Testa J. The Thomson Scientific Journal Selection Process. http://
www.scientific.thomson.com/free/essays/selectionofmaterial/
journalselection/.
Garfield E. The significant scientific literature appears in a small
core of journals. Scientist 1996; 10 (17): 13.
Seglen PO. Why the impact factor of journals should not be used
for evaluating research. BMJ 1997; 314: 498–502.
ISI Web of Knowledge. Journal Citation Reports. Philadelphia:
Thomson Scientific, 2006.
Gluud LL, Sørensen TIA, Gøtzsche PC, Gluud C. The journal impact factor as a predictor of trial quality and outcomes: cohort
study of hepatobiliary randomised clinical trials. Am J Gastroenterol 2005; 100: 2431–2435.
Lee KP, Schotland M, Bacchetti P, Bero LA. Association of journal quality indicators with methodological quality of clinical research articles. JAMA 2002; 287: 2805–2808.
Opthof T. Sense and nonsense about the impact factor. Cardiovasc
Res 1997; 33: 1–7.
Not-so-deep impact. Nature 2005; 435: 1003–1004.
Hecht F, Hecht BK, Sandberg AA. The journal ‘‘impact factor’’:
a misnamed, misleading, misused measure. Cancer Genet Cytogenet 1998; 104: 77–81.
Kurmis AP. Understanding the limitations of the journal impact
factor. J Bone Joint Surg 2003; 85-A: 2449–2454.
Smith R. Measuring the societal impact of research. BMJ 2001;
323: 528.
Maier M. Men’s health and gender medicine: scientific impact or
societal impact? J Mens Health Gend 2006; 3: 330–331.
Kalucy L, McIntyre E, Jackson Bowers E. Primary Health Care
Research Impact Project. Final Report Stage 1. Flinders
University,
http://www.phcris.org.au/phplib/filedownload.
php?file=/elib/lib/downloaded_files/publications/pdfs/phcris_
pub_3338.pdf (accessed on 27 September 2007).
Woolf SH, Johnson RE. The break-even point: when medical advances are less important that improving the fidelity with which
they are delivered. Ann Fam Med 2005; 3: 545–552.