research-article

Public Access

Visualizing Research Impact through Citation Data

Authors:

Huamin QuAuthors Info & Claims

ACM Transactions on Interactive Intelligent Systems (TiiS), Volume 8, Issue 1

Article No.: 5, Pages 1 - 24

https://doi.org/10.1145/3132744

Published: 09 March 2018 Publication History

Abstract

Research impact plays a critical role in evaluating the research quality and influence of a scholar, a journal, or a conference. Many researchers have attempted to quantify research impact by introducing different types of metrics based on citation data, such as h-index, citation count, and impact factor. These metrics are widely used in the academic community. However, quantitative metrics are highly aggregated in most cases and sometimes biased, which probably results in the loss of impact details that are important for comprehensively understanding research impact. For example, which research area does a researcher have great research impact on? How does the research impact change over time? How do the collaborators take effect on the research impact of an individual? Simple quantitative metrics can hardly help answer such kind of questions, since more detailed exploration of the citation data is needed. Previous work on visualizing citation data usually only shows limited aspects of research impact and may suffer from other problems including visual clutter and scalability issues. To fill this gap, we propose an interactive visualization tool, ImpactVis, for better exploration of research impact through citation data. Case studies and in-depth expert interviews are conducted to demonstrate the effectiveness of ImpactVis.

References

[1]

Benjamin Bach, Emmanuel Pietriga, and Ilaria Liccardi. 2013. Visualizing populated ontologies with OntoTrix. International Journal on Semantic Web and Information Systems 9, 4 (Oct. 2013), 17--40.

Digital Library

[2]

Alex Ball and Monica Duke. 2015. How to track the impact of research data with metrics. DCC How-to Guides (2015). http://www.dcc.ac.uk/resources/how-guides/track-data-impact-metrics.

[3]

Pablo D. Batista, Mônica G. Campiteli, and Osame Kinouchi. 2006. Is it possible to compare researchers with different scientific interests? Scientometrics 68, 1 (2006), 179--189.

[4]

Anastasia Bezerianos, Pierre Dragicevic, Jean-Daniel Fekete, Juhee Bae, and Ben Watson. 2010. GeneaQuilts: A system for exploring large genealogies. IEEE Transactions on Visualization and Computer Graphics 16, 6 (Nov. 2010), 1073--1081.

Digital Library

[5]

Renaud Blanch, Rémy Dautriche, and Gilles Bisson. 2015. Dendrogramix: A hybrid tree-matrix visualization technique to support interactive exploration of dendrograms. In Proceedings of the 2015 IEEE Pacific Visualization Symposium (PacificVis). 31--38.

[6]

M. Bostock, V. Ogievetsky, and J. Heer. 2011. D3 data-driven documents. IEEE Transactions on Visualization and Computer Graphics 17, 12 (Dec. 2011), 2301--2309.

Digital Library

[7]

Nan Cao, Yu-Ru Lin, Fan Du, and Dashun Wang. 2016. Episogram: Visual summarization of egocentric social interactions. IEEE Computer Graphics and Applications 36, 5 (Sept. 2016), 72--81.

[8]

Marian Dörk, Nathalie Henry Riche, Gonzalo Ramos, and Susan Dumais. 2012. PivotPaths: Strolling through faceted information spaces. IEEE Transactions on Visualization and Computer Graphics 18, 12 (Dec. 2012), 2709--2718.

Digital Library

[9]

Fan Du, Catherine Plaisant, Neil Spring, and Ben Shneiderman. 2016. EventAction: Visual analytics for temporal event sequence recommendation. In 2016 IEEE Conference on Visual Analytics Science and Technology (VAST). 61--70.

[10]

Leo Egghe. 2006. Theory and practice of the g-index. Scientometrics 69, 1 (2006), 131--152.

[11]

Niklas Elmqvist, Thanh-Nghi Do, Howard Goodell, Nathalie Henry, and Jean-Daniel Fekete. 2008. ZAME: Interactive large-scale graph visualization. In 2008 IEEE Pacific Visualization Symposium. 215--222.

[12]

China Computer Federation. 2005. A list of computer science conferences and journals recommended by CCF. (2005). Retrieved Oct. 27, 2016 from http://history.ccf.org.cn/sites/ccf/paiming.jsp.

[13]

Jean-Daniel Fekete, Niklas Elmqvist, Thanh-Nghi Do, Howard Goodell, and Nathalie Henry. 2007. Navigating Wikipedia with the Zoomable Adjacency Matrix Explorer. Research Report RR-6163. INRIA. 25 pages. https://hal.inria.fr/inria-00141168.

[14]

Jane Grimson. 2014. Measuring research impact: Not everything that can be counted counts, and not everything that counts can be counted. Bibliometrics. Use and Abuse in the Review of Research Performance (2014), 29--41.

[15]

Jeffrey Heer, Stuart K. Card, and James A. Landay. 2005. Prefuse: A toolkit for interactive information visualization. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’05). ACM, New York, NY, 421--430.

Digital Library

[16]

Florian Heimerl, Qi Han, Steffen Koch, and Thomas Ertl. 2016. CiteRivers: Visual analytics of citation patterns. IEEE Transactions on Visualization and Computer Graphics 22, 1 (Jan. 2016), 190--199.

Digital Library

[17]

Nathalie Henry, Jean-Daniel Fekete, and Michael J. McGuffin. 2007. NodeTrix: A hybrid visualization of social networks. IEEE Transactions on Visualization and Computer Graphics 13, 6 (Nov. 2007), 1302--1309.

Digital Library

[18]

Jorge E. Hirsch. 2005. An index to quantify an individual’s scientific research output. Proceedings of the National Academy of Sciences of the United States of America 102, 46 (2005), 16569--16572.

[19]

Jasleen Kaur, Filippo Radicchi, and Filippo Menczer. 2013. Universality of scholarly impact metrics. Journal of Informetrics 7, 4 (2013), 924--932.

[20]

Kwon Kyeongan, Majeti Dinesh, Uzzi Brian, and Ioannis Pavlidis. 2016. Scholar plot. (May 2016). Retrieved Sept. 12, 2016 from http://scholarplot.com/index.html.

[21]

Bongshin Lee, Mary Czerwinski, George Robertson, and Benjamin B. Bederson. 2005. Understanding research trends in conferences using paperLens. In CHI’05 Extended Abstracts on Human Factors in Computing Systems (CHI EA’05). ACM, New York, NY, 1969--1972.

Digital Library

[22]

Loet Leydesdorff and Lutz Bornmann. 2011. Integrated impact indicators compared with impact factors: An alternative research design with policy implications. Journal of the American Society for Information Science and Technology 62, 11 (2011), 2133--2146.

Digital Library

[23]

Evelyne Lutton, Hugo Gilbert, Waldo Cancino, Benjamin Bach, Pierre Parrend, and Pierre Collet. 2014. Gridvis: Visualisation of island-based parallel genetic algorithms. In European Conference on the Applications of Evolutionary Computation. Springer Berlin, Berlin, Germany, 702--713.

[24]

David J. Lynn, Geoffrey L. Winsor, Calvin Chan, Nicolas Richard, Matthew R. Laird, Aaron Barsky, Jennifer L. Gardy, Fiona M. Roche, Timothy H. W. Chan, Naisha Shah, and others. 2008. InnateDB: Facilitating systems-level analyses of the mammalian innate immune response. Molecular Systems Biology 4, 1 (2008), 218.

[25]

Ian Scott MacKenzie. 2009. Citedness, uncitedness, and the murky world between. In CHI’09 Extended Abstracts on Human Factors in Computing Systems (CHI EA’09). ACM, New York, NY, 2545--2554.

Digital Library

[26]

Eamonn Maguire, Javier Martin Montull, and Gilles Louppe. 2016. Visualization of publication impact. In Proceedings of the Eurographics/IEEE VGTC Conference on Visualization: Short Papers (EuroVis’16). Eurographics Association, Goslar, Germany, 103--107.

Digital Library

[27]

Justin Matejka, Tovi Grossman, and George Fitzmaurice. 2012. Citeology: Visualizing paper genealogy. In CHI’12 Extended Abstracts on Human Factors in Computing Systems (CHI EA’12). ACM, New York, NY, 181--190.

Digital Library

[28]

Teresa Penfield, Matthew J. Baker, Rosa Scoble, and Michael C. Wykes. 2014. Assessment, evaluations, and definitions of research impact: A review. Research Evaluation 23, 1 (2014), 21--32.

[29]

Adam Perer and Jimeng Sun. 2012. MatrixFlow: Temporal network visual analytics to track symptom evolution during disease progression. In AMIA Annual Symposium Proceedings, Vol. 2012. American Medical Informatics Association, 716--725.

[30]

Filippo Radicchi, Santo Fortunato, and Claudio Castellano. 2008. Universality of citation distributions: Toward an objective measure of scientific impact. Proceedings of the National Academy of Sciences of the United States of America 105, 45 (2008), 17268--17272.

[31]

Lei Shi, Hanghang Tong, Jie Tang, and Chuang Lin. 2015. Vegas: Visual influence graph summarization on citation networks. IEEE Transactions on Knowledge and Data Engineering 27, 12 (Dec. 2015), 3417--3431.

Digital Library

[32]

Ben Shneiderman. 1996. The eyes have it: A task by data type taxonomy for information visualizations. In Proceedings of the 1996 IEEE Symposium on Visual Languages. 336--343.

Digital Library

[33]

Arnab Sinha, Zhihong Shen, Yang Song, Hao Ma, Darrin Eide, Bo-June (Paul) Hsu, and Kuansan Wang. 2015. An overview of Microsoft academic service (MAS) and applications. In Proceedings of the 24th International Conference on World Wide Web. ACM, New York, NY, 243--246.

Digital Library

[34]

Jonathan Stallings, Eric Vance, Jiansheng Yang, Michael W. Vannier, Jimin Liang, Liaojun Pang, Liang Dai, Ivan Ye, and Ge Wang. 2013. Determining scientific impact using a collaboration index. Proceedings of the National Academy of Sciences of the United States of AMerica 110, 24 (2013), 9680--9685.

[35]

John Stasko, Jaegul Choo, Yi Han, Mengdie Hu, Hannah Pileggi, Ramik Sadana, and Charles D. Stolper. 2013. Citevis: Exploring conference paper citation data visually. Posters of IEEE Information Visualization (2013).

[36]

Jie Tang, Jing Zhang, Limin Yao, Juanzi Li, Li Zhang, and Zhong Su. 2008. ArnetMiner: Extraction and mining of academic social networks. In Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD’08). ACM, New York, NY, 990--998.

Digital Library

[37]

Robert J. W. Tijssen. 2003. Scoreboards of research excellence. Research Evaluation 12, 2 (2003), 91--103.

[38]

Tatiana Von Landesberger, Arjan Kuijper, Tobias Schreck, Jörn Kohlhammer, Jarke J. van Wijk, J.-D. Fekete, and Dieter W. Fellner. 2011. Visual analysis of large graphs: State-of-the-art and future research challenges. In Computer Graphics Forum, Vol. 30. Wiley Online Library, 1719--1749.

[39]

Colin Ware. 2012. Information Visualization: Perception for Design (3rd ed.). Morgan Kaufmann, San Francisco, CA.

Digital Library

[40]

Meng Qi Yelena Wu, Robert Faris, and Kwan-Liu Ma. 2013. Visual exploration of academic career paths. In Proceedings of the 2013 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM’13). ACM, New York, NY, 779--786.

Digital Library

[41]

Lexing Xie. 2016. Visualizing citation patterns of computer science conferences. (Aug. 2016). Retrieved Sept. 16, 2016 from http://cm.cecs.anu.edu.au/post/citation_vis.

[42]

Panpan Xu, Nan Cao, Huamin Qu, and John Stasko. 2016. Interactive visual co-cluster analysis of bipartite graphs. In 2016 IEEE Pacific Visualization Symposium (PacificVis). 32--39.

[43]

Wei Xu, Xin Liu, and Yihong Gong. 2003. Document clustering based on non-negative matrix factorization. In Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’03). ACM, New York, NY, 267--273.

Digital Library

[44]

Jian Zhao, Christopher Collins, Fanny Chevalier, and Ravin Balakrishnan. 2013. Interactive exploration of implicit and explicit relations in faceted datasets. IEEE Transactions on Visualization and Computer Graphics 19, 12 (Dec. 2013), 2080--2089.

Digital Library

[45]

Jian Zhao, Michael Glueck, Fanny Chevalier, Yanhong Wu, and Azam Khan. 2016. Egocentric analysis of dynamic networks with egoLines. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI’16). ACM, New York, NY, 5003--5014.

Digital Library

[46]

Jian Zhao, Zhicheng Liu, Mira Dontcheva, Aaron Hertzmann, and Alan Wilson. 2015. MatrixWave: Visual comparison of event sequence data. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI’15). ACM, New York, NY, 259--268.

Digital Library

Cited By

Beck F(2024)PUREsuggest: Citation-Based Literature Search and Visual Exploration with Keyword-Controlled RankingsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.345619931:1(316-326)Online publication date: 10-Sep-2024
https://dl.acm.org/doi/10.1109/TVCG.2024.3456199
(2024)BibliographyHuman-Computer Interaction10.1016/B978-0-44-314096-9.00015-4(391-429)Online publication date: 2024
https://doi.org/10.1016/B978-0-44-314096-9.00015-4
MacKenzie I(2024)Scientific foundationsHuman-Computer Interaction10.1016/B978-0-44-314096-9.00010-5(161-201)Online publication date: 2024
https://doi.org/10.1016/B978-0-44-314096-9.00010-5
Show More Cited By

Index Terms

Visualizing Research Impact through Citation Data
1. Human-centered computing
  1. Visualization
    1. Visualization application domains
      1. Information visualization
    2. Visualization design and evaluation methods

Recommendations

Altmetrics and citation counts: an empirical analysis of the computer science domain
JCDL '22: Proceedings of the 22nd ACM/IEEE Joint Conference on Digital Libraries

Background. Researchers, funding agencies, and institutions involve bibliographic data to assess the impact or reputation of papers, publication venues, researchers, and institutions. Particularly citation counts, and metrics that build on these (e.g., ...
Investigating different types of research collaboration and citation impact: a case study of Harvard University's publications

This study aims to investigate the influence of different patterns of collaboration on the citation impact of Harvard University's publications. Those documents published by researchers affiliated with Harvard University in WoS from 2000---2009, ...
A combined bibliometric indicator to predict article impact

In both the UK and Australia there has been a recent move to use citation analysis in the evaluation of the research of individuals. In particular, the future UK Research Excellence Framework (REF), proposes using citation data in the research ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Interactive Intelligent Systems

ACM Transactions on Interactive Intelligent Systems Volume 8, Issue 1

Special Issue on Interactive Visual Analysis of Human Crowd Behaviors and Regular Paper

March 2018

132 pages

ISSN:2160-6455

EISSN:2160-6463

DOI:10.1145/3185338

Editor:
Michelle Zhou
Juji, Inc., USA

Issue’s Table of Contents

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 09 March 2018

Accepted: 01 July 2017

Revised: 01 May 2017

Received: 01 January 2017

Published in TIIS Volume 8, Issue 1

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

Baidu gift
NIH
ARO
DTRA
National 973 Program of China
RGC GRF
NSF

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
677
Total Downloads

Downloads (Last 12 months)112
Downloads (Last 6 weeks)16

Reflects downloads up to 31 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Beck F(2024)PUREsuggest: Citation-Based Literature Search and Visual Exploration with Keyword-Controlled RankingsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.345619931:1(316-326)Online publication date: 10-Sep-2024
https://dl.acm.org/doi/10.1109/TVCG.2024.3456199
(2024)BibliographyHuman-Computer Interaction10.1016/B978-0-44-314096-9.00015-4(391-429)Online publication date: 2024
https://doi.org/10.1016/B978-0-44-314096-9.00015-4
MacKenzie I(2024)Scientific foundationsHuman-Computer Interaction10.1016/B978-0-44-314096-9.00010-5(161-201)Online publication date: 2024
https://doi.org/10.1016/B978-0-44-314096-9.00010-5
Li QWang XFu LCao XWang XZhang JZhou C(2024)VSAN: A new visualization method for super-large-scale academic networksFrontiers of Computer Science: Selected Publications from Chinese Universities10.1007/s11704-022-2078-518:1Online publication date: 1-Feb-2024
https://dl.acm.org/doi/10.1007/s11704-022-2078-5
Kale BSun MPapka M(2023)The State of the Art in Visualizing Dynamic Multivariate NetworksComputer Graphics Forum10.1111/cgf.1485642:3(471-490)Online publication date: 27-Jun-2023
https://doi.org/10.1111/cgf.14856
Guo ZTao JChen SChawla NWang C(2023) SD 2 : Slicing and Dicing Scholarly Data for Interactive Evaluation of Academic Performance IEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2022.316372729:8(3569-3585)Online publication date: 1-Aug-2023
https://doi.org/10.1109/TVCG.2022.3163727
Tian MLi GYuan X(2023)LitVis: a visual analytics approach for managing and exploring literatureJournal of Visualization10.1007/s12650-023-00941-326:6(1445-1458)Online publication date: 1-Dec-2023
https://dl.acm.org/doi/10.1007/s12650-023-00941-3
Wang YPeng TLu HWang HXie XQu HWu Y(2022)Seek for Success: A Visualization Approach for Understanding the Dynamics of Academic CareersIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2021.311479028:1(475-485)Online publication date: 1-Jan-2022
https://dl.acm.org/doi/10.1109/TVCG.2021.3114790
Sun LZhang XPan XLiu YYu WXu TLiu FChen WWang YSu WZhou Z(2022)Visual analytics of genealogy with attribute-enhanced topological clusteringJournal of Visualization10.1007/s12650-021-00802-x25:2(361-377)Online publication date: 1-Apr-2022
https://dl.acm.org/doi/10.1007/s12650-021-00802-x
Shin MSoen AReadshaw BBlackburn SWhitelaw MXie L(2019)Influence Flowers of Academic Entities2019 IEEE Conference on Visual Analytics Science and Technology (VAST)10.1109/VAST47406.2019.8986934(1-10)Online publication date: Oct-2019
https://doi.org/10.1109/VAST47406.2019.8986934

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Figures

Tables

Media

View Issue’s Table of Contents