note

Evaluating Information Visualization via the Interplay of Heuristic Evaluation and Question-Based Scoring

Authors:

Marti A. Hearst,

Paul Laskowski,

Luis SilvaAuthors Info & Claims

CHI '16: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems

Pages 5028 - 5033

https://doi.org/10.1145/2858036.2858280

Published: 07 May 2016 Publication History

Abstract

In an instructional setting it can be difficult to accurately assess the quality of information visualizations of several variables. Instead of a standard design critique, an alternative is to ask potential readers of the chart to answer questions about it. A controlled study with 47 participants shows a good correlation between aggregated novice heuristic evaluation scores and results of answering questions about the data, suggesting that the two forms of assessment can be complementary. Using both metrics in parallel can yield further benefits; discrepancies between them may reveal incorrect application of heuristics or other issues.

Supplementary Material

ZIP File (pn1277-file4.zip)

pn1277-file4.zip

Download
2.34 MB

References

[1]

Richard I Anderson, Jeremy Ashley, Tobias Herrmann, Justin Miller, Jim Nieters, Shauna Sampson Eves, and Secil Tabli Watson. 2007. Moving UX into a position of corporate influence: whose advice really works?. In CHI'07 Extended Abstracts on Human Factors in Computing Systems. ACM, 1905--1908.

Digital Library

[2]

Jacques Bertin. 1983. Semiology of graphics: diagrams, networks, maps. University of Wisconsin press.

Digital Library

[3]

Alberto Cairo. 2012. The Functional Art: An introduction to information graphics and visualization. New Riders.

Digital Library

[4]

Sheelagh Carpendale. 2008. Evaluating information visualizations. In Information Visualization. Springer, 19--45.

Digital Library

[5]

William S Cleveland and Robert McGill. 1984. Graphical perception: Theory, experimentation, and application to the development of graphical methods. Journal of the American statistical association 79, 387 (1984), 531--554.

[6]

Heather Desurvire, Jim Kondziela, and Michael E Atwood. 1992. What is gained and lost when using methods other than empirical testing. In Posters and short talks of the 1992 SIGCHI conference on Human factors in computing systems. ACM, 125--126.

Digital Library

[7]

Stephen Few. 2006. Visual Communication: Design Principles for Displaying Quantitative Information. Cognos Innovation Center (September 2006). http://www.perceptualedge.com/articles/Whitepapers/ Visual_Communication.pdf.

[8]

Stephen Few. 2009. Now you see it: simple visualization techniques for quantitative analysis. Analytics Press.

Digital Library

[9]

Yoav Freund and Robert E Schapire. 1996. Experiments with a new boosting algorithm. In ICML, Vol. 96. 148--156.

Digital Library

[10]

Marti A Hearst, Eytan Adar, Robert Kosara, Tamara Munzner, Jon Schwabish, and Ben Shneiderman. 2015. Vis, The Next Generation: Teaching Across the Researcher-Practitioner Gap. In IEEE Conference on Information Visualization {Panel}.

[11]

Jeffrey Heer and Michael Bostock. 2010. Crowdsourcing graphical perception: using mechanical turk to assess visualization design. In CHI. ACM, 203--212.

Digital Library

[12]

Tasha Hollingsed and David G Novick. 2007. Usability inspection methods after 15 years of research and practice. In Proceedings of the 25th Annual ACM international conference on Design of communication. ACM, 249--255.

Digital Library

[13]

Tobias Isenberg, Petra Isenberg, Jian Chen, Michael Sedlmair, and Torsten Moller. 2013. A systematic review on the practice of evaluating visualization. Visualization and Computer Graphics, IEEE Transactions on 19, 12 (2013), 2818--2827.

Digital Library

[14]

Robin Jeffries and Heather Desurvire. 1992. Usability testing vs. heuristic evaluation: was there a contest? ACM SIGCHI Bulletin 24, 4 (1992), 39--41.

Digital Library

[15]

Robin Jeffries, James R Miller, Cathleen Wharton, and Kathy Uyeda. 1991. User interface evaluation in the real world: a comparison of four techniques. In Proceedings of the SIGCHI conference on Human factors in computing systems. ACM, 119--124.

Digital Library

[16]

Ron Kohavi, Roger Longbotham, Dan Sommerfield, and Randal M. Henne. 2009. Controlled experiments on the web: survey and practical guide. Data Min. Knowl. Discov. 18 (2009), 140--181.

Digital Library

[17]

Robert Kosara and Caroline Ziemkiewicz. 2010. Do Mechanical Turks dream of square pie charts?. In Proceedings of the 3rd BELIV'10 Workshop: BEyond time and errors: novel evaLuation methods for Information Visualization. ACM, 63--70.

Digital Library

[18]

Chinmay Kulkarni, Koh Pang Wei, Huy Le, Daniel Chia, Kathryn Papadopoulos, Justin Cheng, Daphne Koller, and Scott R Klemmer. 2013. Peer and self assessment in massive online classes. In ACM Transactions on Computer Human Interaction (TOCHI). Vol. 20. ACM.

Digital Library

[19]

Heidi Lam, Enrico Bertini, Petra Isenberg, Catherine Plaisant, and Sheelagh Carpendale. 2012. Empirical studies in information visualization: Seven scenarios. Visualization and Computer Graphics, IEEE Transactions on 18, 9 (2012), 1520--1536.

Digital Library

[20]

Gitte Lindgaard and Jarinee Chattratichart. 2007. Usability testing: what have we overlooked?. In Proceedings of the SIGCHI Conference on Human factors in Computing Systems. ACM, 1415--1424.

Digital Library

[21]

Kurt Luther, J Tolentino, Wei Wu, Amy Pavel, B Bailey, Maneesh Agrawala, Björn Hartmann, and Steven P Dow. 2015. Structuring, Aggregating, and Evaluating Crowdsourced Design Critique. CSCW.

Digital Library

[22]

Jock Mackinlay. 1988. Applying a theory of graphical presentation to the graphic design of user interfaces. In UIST. ACM, 179--189.

Digital Library

[23]

Jakob Nielsen. 1992. Finding usability problems through heuristic evaluation. In Proceedings of the SIGCHI conference on Human factors in computing systems. ACM, 373--380.

Digital Library

[24]

Jakob Nielsen and Thomas K Landauer. 1993. A mathematical model of the finding of usability problems. In Proceedings of the INTERACT'93 and CHI'93 conference on Human factors in computing systems. ACM, 206--213.

Digital Library

[25]

Jakob Nielsen and Rolf Molich. 1990. Heuristic evaluation of user interfaces. In CHI. ACM, 249--256.

Digital Library

[26]

Chris North. 2006. Toward measuring visualization insight. Computer Graphics and Applications, IEEE 26, 3 (2006), 6--9.

Digital Library

[27]

Lucy Nowell, Robert Schulman, and Deborah Hix. 2002. Graphical encoding for information visualization: an empirical study. In Information Visualization, 2002. INFOVIS 2002. IEEE Symposium on. IEEE, 43--50.

Digital Library

[28]

David A Robb, Stefano Padilla, Britta Kalkreuter, and Mike J Chantler. 2015. Crowdsourced Feedback With Imagery Rather Than Text: Would Designers Use It?. In CHI. ACM, 1355--1364.

Digital Library

[29]

James H Steiger. 2004. Beyond the F test: Effect size confidence intervals and tests of close fit in the analysis of variance and contrast analysis. Psychological methods 9, 2 (2004), 164.

[30]

David Tinapple, Loren Olson, and John Sadauskas. 2013. CritViz: Web-based software supporting peer critique in large creative classrooms. Bulletin of the IEEE Technical Committee on Learning Technology 15, 1 (2013), 29.

[31]

Colin Ware. 2012. Information visualization: perception for design. Elsevier.

Digital Library

[32]

Huahai Yang, Yunyao Li, and Michelle X Zhou. 2014. Understand users' comprehension and preferences for composing information visualizations. ACM Transactions on Computer-Human Interaction (TOCHI) 21, 1 (2014), 6.

Digital Library

[33]

Alvin Yuan, Kurt Luther, Markus Krause, Sophie Vennix, Stephen P. Dow, and Björn Hartmann. 2016. Almost an Expert: The Effects of Rubrics and Expertise on Perceived Value of Crowdsourced Design Critiques. CSCW. to appear.

Digital Library

[34]

Ying Zhu. 2007. Measuring effective data visualization. Advances in Visual Computing, Lecture Notes in Computer Science (LNCS) 4842 (2007), 652--661.

Digital Library

Cited By

Errey NZowghi DLeong TWu YYuan XHuang WLiang C(2025)Heuristics for evaluating narrative visualization: a validation studyJournal of Visualization10.1007/s12650-025-01051-yOnline publication date: 24-Feb-2025
https://doi.org/10.1007/s12650-025-01051-y
Burns ALee CChawla RPeck EMahyar N(2023)Who Do We Mean When We Talk About Visualization Novices?Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581524(1-16)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581524
Choi JOh CKim YKim N(2023)VisLab: Enabling Visualization Designers to Gather Empirically Informed Design FeedbackProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581132(1-18)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581132
Show More Cited By

Index Terms

Evaluating Information Visualization via the Interplay of Heuristic Evaluation and Question-Based Scoring
1. Human-centered computing
  1. Collaborative and social computing
    1. Collaborative and social computing systems and tools
2. Information systems
  1. World Wide Web

Recommendations

Finding usability problems through heuristic evaluation
CHI '92: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

Usability specialists were better than non-specialists at performing heuristic evaluation, and “double experts” with specific expertise in the kind of interface being evaluated performed even better. Major usability problems have a higher probability ...
An heuristic set for evaluation in information visualization
AVI '10: Proceedings of the International Conference on Advanced Visual Interfaces

Evaluation is a key research challenge within the international Information Visualization (InfoVis) community, and Heuristic Evaluation is one recognized method. Various sets of heuristics have been proposed but there remains no consensus as to which ...
Evaluation of Visualization Heuristics
Human-Computer Interaction. Theories, Methods, and Human Issues
Abstract
Multiple sets of heuristic have been developed and studied in the Human Computer Interaction (HCI) domain as a method for fast, lightweight evaluations for usability problems. However, none of the heuristics have been adopted by the information ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '16: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems

May 2016

6108 pages

ISBN:9781450333627

DOI:10.1145/2858036

General Chairs:
Jofish Kaye
Yahoo
,
Allison Druin
University of Maryland / National Park Service
,
Program Chairs:
Cliff Lampe
University of Michigan
,
Dan Morris
Microsoft
,
Juan Pablo Hourcade
University of Iowa

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 May 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Note

Funding Sources

Google

Conference

CHI'16

Sponsor:

SIGCHI

CHI'16: CHI Conference on Human Factors in Computing Systems

May 7 - 12, 2016

California, San Jose, USA

Acceptance Rates

CHI '16 Paper Acceptance Rate 565 of 2,435 submissions, 23%;

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI 2025

Sponsor:
sigchi

ACM CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

22
Total Citations
View Citations
693
Total Downloads

Downloads (Last 12 months)31
Downloads (Last 6 weeks)0

Reflects downloads up to 03 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Errey NZowghi DLeong TWu YYuan XHuang WLiang C(2025)Heuristics for evaluating narrative visualization: a validation studyJournal of Visualization10.1007/s12650-025-01051-yOnline publication date: 24-Feb-2025
https://doi.org/10.1007/s12650-025-01051-y
Burns ALee CChawla RPeck EMahyar N(2023)Who Do We Mean When We Talk About Visualization Novices?Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581524(1-16)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581524
Choi JOh CKim YKim N(2023)VisLab: Enabling Visualization Designers to Gather Empirically Informed Design FeedbackProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581132(1-18)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581132
Lee-Robbins EHe SAdar E(2022)Learning Objectives, Insights, and Assessments: How Specification Formats Impact DesignIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2021.311481128:1(676-685)Online publication date: 1-Jan-2022
https://dl.acm.org/doi/10.1109/TVCG.2021.3114811
Heine C(2021)Towards Modeling Visualization Processes as Dynamic Bayesian NetworksIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2020.303039527:2(1000-1010)Online publication date: Feb-2021
https://doi.org/10.1109/TVCG.2020.3030395
Adar ELee E(2021)Communicative Visualizations as a Learning ProblemIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2020.303037527:2(946-956)Online publication date: Feb-2021
https://doi.org/10.1109/TVCG.2020.3030375
Ye YSauer FMa KAditya KChen J(2020)A User-Centered Design Study in Scientific Visualization Targeting Domain ExpertsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2020.297052526:6(2192-2203)Online publication date: 1-Jun-2020
https://doi.org/10.1109/TVCG.2020.2970525
Beasley ZFriedman APieg LRosen P(2020)Leveraging Peer Feedback to Improve Visualization Education2020 IEEE Pacific Visualization Symposium (PacificVis)10.1109/PacificVis48177.2020.1261(146-155)Online publication date: Jun-2020
https://doi.org/10.1109/PacificVis48177.2020.1261
de Souza Alencar WAbdala Rfaei Jradi WDantas do Nascimento HFelix JAlves de Melo Nunes Soares F(2020)An Empirical Methodological Study of Evaluation Methods Applied to Educational Timetabling VisualizationsAdvances in Visual Computing10.1007/978-3-030-64556-4_17(209-223)Online publication date: 7-Dec-2020
https://doi.org/10.1007/978-3-030-64556-4_17
Mylly SRajanen MIivari N(2020)The Quest for Usable Usability Heuristics for Game DevelopersAdvances in Information Systems Development10.1007/978-3-030-49644-9_10(164-181)Online publication date: 1-Aug-2020
https://doi.org/10.1007/978-3-030-49644-9_10
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten