Abstract
The present paper addresses some of the many possible uses of citations, including bookmark, intellectual heritage, impact tracker, and self-serving purposes. The main focus is on the applicability of citation analysis as an impact or quality measure. If a paper's bibliography is viewed as consisting of a directed (research impact or quality) component related to intellectual heritage and random components related to specific self-interest topics, then for large numbers of citations from many different citing paper, the most significant intellectual heritage (research impact or quality) citations will aggregate and the random author-specific self-serving citations will be scattered and not accumulate. However, there are at least two limitations to this model of citation analysis for stand-alone use as a measure of research impact of quality. First, the reference to intellectual heritage could be positive or negative. Second, there could be systemic biases which affect the aggregate results, and one of these, the “Pied Piper Effect”, is described in detail. Finally, the results of a short citation study comparing Russian and American papers in different technical fields are presented. The questions raised in interpreting this data highlight a few of the difficulties in attempting to interpret citation results without supplementary information.
Leydesdorff (Leydesdorff, 1998) addresses the history of citations and citation analysis, and the transformation of a reference mechanism into a purportedly quantitive measure of research impact/quality. The present paper examines different facets of citations and citation analysis, and discusses the validity of citation analysis as a useful measure of research impact/quality.
Similar content being viewed by others
References
Kostoff, R. N. (1994), Research impact quantification,R&D Management, 24: 3, July.
Kostoff, R. N. (1995). Federal research impact assessment: Axioms, approaches, applications,Scientometrics, 34: 2.
Kostoff, R. N. (1997a),The Handbook of Research Impact Assessment, Seventh Edition. DTIC Report Number ADA296021. Also, available at http://www.dtic.mil/dtic/kostoff/index.html
Kostoff, R. N. (1997b), Peer review: The appropriate GPRA metric for research,Science, Vol. 277. 1 August. p. 651–652.
Kostoff, R. N. (1997c), Use and misuse of metrics in research evaluation,Science and Engineering Ethics, 3:2.
Kostoff, R. N. (1989a), Hypersonic and supersonic flow roadmaps using bibliometrics and database tomography, (submitted for publication).
Kostoff, R. N. (1998b), Database tomography for technical intelligence: A roadmap of the near-earth space science and technology literature,Information Processing and Management, 34:1.
Leydesdorff, L. (1998), Theories of Citation,Scientometrics, 43 (1) 5–25.
Macroberts, M. andMacroberts, B. (1996), Problems of citation analysis,Scientometrics, 36 (3) 435. Top 10 U. S. universities in clinical medicine research, 1990–1994,Science. Vol. 269. 1 September, 1995. p. 1223.
Scientists who fund themselves,Science, Vol. 279. 9 January, 1998, p. 179.
Author information
Authors and Affiliations
Additional information
Comments on Theories of Citation?L. Leydesdorff,Scientometrics,43 (1998) No. 1.
Rights and permissions
About this article
Cite this article
Kostoff, R.N. The use and misuse of citation analysis in research evaluation. Scientometrics 43, 27–43 (1998). https://doi.org/10.1007/BF02458392
Received:
Issue Date:
DOI: https://doi.org/10.1007/BF02458392