research-article

An Insight-Based Methodology for Evaluating Bioinformatics Visualizations

Authors:

Purvi Saraiya,

Chris North,

Karen DucaAuthors Info & Claims

IEEE Transactions on Visualization and Computer Graphics, Volume 11, Issue 4

Pages 443 - 456

https://doi.org/10.1109/TVCG.2005.53

Published: 01 July 2005 Publication History

Publisher Site

Abstract

High-throughput experiments, such as gene expression microarrays in the life sciences, result in very large data sets. In response, a wide variety of visualization tools have been created to facilitate data analysis. A primary purpose of these tools is to provide biologically relevant insight into the data. Typically, visualizations are evaluated in controlled studies that measure user performance on predetermined tasks or using heuristics and expert reviews. To evaluate and rank bioinformatics visualizations based on real-world data analysis scenarios, we developed a more relevant evaluation method that focuses on data insight. This paper presents several characteristics of insight that enabled us to recognize and quantify it in open-ended user tests. Using these characteristics, we evaluated five microarray visualization tools on the amount and types of insight they provide and the time it takes to acquire it. The results of the study guide biologists in selecting a visualization tool based on the type of their microarray data, visualization designers on the key role of user interaction techniques, and evaluators on a new approach for evaluating the effectiveness of visualizations for providing insight. Though we used the method to analyze bioinformatics visualizations, it can be applied to other domains.

References

[1]

D. Duggan B. Bittner Y. Chen P. Meltzer and J. Trent, “Expression Profiling Using cDNA Microarrays,” Nature Genetics, vol. 21, pp. 11-19, Jan. 1999.

Crossref

Google Scholar

[2]

L. Shi, “DNA Microarray-Genome Chip,” http://www.gene-chips.com/GeneChips.html#What, 2002.

Google Scholar

[3]

D. Bassett M. Eisen and M. Boguski, “Gene Expression Informatics-Its All in Your Mine,” Nature Genetics Supplement, vol. 21, Jan. 1999.

Google Scholar

[4]

N. Bolshakova, “Microarray Software Catalogue,” http://www.cs.tcd.ie/Nadia.Bolshakova/softwaretotal.html, 2004.

Google Scholar

[5]

Y. Leung, “Functional Genomics,” http://genomicshome.com, 2004.

Google Scholar

[6]

A. Robinson, “Bioinformatics Visualization,” http://industry.ebi. ac.uk/alan/, 2002.

Google Scholar

[7]

R. Spence, Information Visualization. Addison-Wesley, 2001.

Google Scholar

[8]

S. Card J.D. Mackinlay and B. Shneiderman, Readings in Information Visualization-Using Visualization to Think. San Francisco: Morgan Kaufmann, 1999.

Digital Library

Google Scholar

[9]

SPOTFIRE® Decision site for functional Genomics, http://www.Spotfire.com, 2005.

Google Scholar

[10]

C. Chen and M. Czerwinski, “Empirical Evaluation of Information Visualizations: An Introduction,” Int'l J. Human-Computer Studies, vol. 53, pp. 631-635, 2000.

Digital Library

Google Scholar

[11]

C. Chen and Y. Yu, “Empirical Studies of Information Visualization: A Meta-Analysis,” Int'l J. Human-Computer Studies, vol. 53, pp. 851-866, 2000.

Digital Library

Google Scholar

[12]

P. Saraiya C. North and K. Duca, “An Evaluation of Microarray Visualization Tools for Biological Insight,” Proc. IEEE InfoVis, 2004.

Digital Library

Google Scholar

[13]

A. Kobsa, “An Empirical Comparison of Three Commercial Information Visualization Systems,” Proc. IEEE InfoVis 2001, pp.nbsp123-130, 2001.

Digital Library

Google Scholar

[14]

P. Irani and C. Ware, “Diagramming Information Structures Using 3D Perceptual Primitives,” ACM Trans. Computer-Human Interaction, vol. 10, no. 1, pp. 1-19, 2003.

Digital Library

Google Scholar

[15]

H. Hartson and D. Hix, Developing User Interfaces: Ensuring Usability through Product and Process. John Wiley, 1993.

Digital Library

Google Scholar

[16]

G. Rao and D. Mingay, “Report on Usability Testing of Census Bureau's Dynamaps CD-ROM Product,” http://infovis.cs.vt. edu/cs5764/papers/dynamapsUsability.pdf, 2001.

Google Scholar

[17]

J. Nielsen, “Finding Usability Problems through Heuristic Evaluation,” Proc. ACM Conf. Computer Human Interaction (CHInbsp'92), pp. 373-380, 1992.

Digital Library

Google Scholar

[18]

J.D. Mackinlay, “Automating the Design of Graphical Presentations of Relational Information,” ACM Trans. Graphics, vol. 5, pp.nbsp110-141, 1986.

Digital Library

Google Scholar

[19]

E. Tufte, The Visual Display of Quantitative Information. Graphics Press, 1983.

Digital Library

Google Scholar

[20]

C. Freitas P. Luzzardi R. Cava M. Pimenta A. Winckler and L. Nedel, “Evaluating Usability of Information Visualization Techniques,” Proc. Advanced Visual Interfaces (AVI '02), pp. 373-374, 2002.

Digital Library

Google Scholar

[21]

B. Shneiderman, “The Eyes Have It: A Task by Data Type Taxonomy,” Proc. IEEE Symp. Visual Languages '96, pp. 336-343, 1996.

Digital Library

Google Scholar

[22]

C. Ware, Information Visualization: Perception for Design. Morgan Kaufmann, 2004.

Digital Library

Google Scholar

[23]

W.S. Cleveland, The Elements of Graphing Data. Hobart Press, 1994.

Digital Library

Google Scholar

[24]

O. Juarez, “CAEVA: Cognitive Architecture to Evaluate Visualization Applications,” Proc. Int'l Conf. Information Visualization (IVnbsp'03), pp. 589-595, 2003.

Digital Library

Google Scholar

[25]

V. Gonzales and A. Kobsa, “A Workplace Study of the Adoption of Information Visualization Systems,” Proc. I-KNOW '03: Third Int'l Conf. Knowledge Management, pp. 92-102, 2003.

Google Scholar

[26]

J. Rieman, “A Field Study of Exploratory Learning Strategies,” ACM Trans. Computer-Human Interaction, vol. 3, pp. 189-218, 1996.

Digital Library

Google Scholar

[27]

J.-D. Fekete and C. Plaisant, Infovis 2003 Contest, http://www.cs. umd.edu/hcil/iv03contest/, 2003.

Google Scholar

[28]

C. Plaisant, “The Challenge of Information Visualization Evaluation,” Proc. Advanced Visual Interfaces (AVI '04), 2004.

Digital Library

Google Scholar

[29]

GENESPRING®, Cutting-Edge Tools for Expression Analysis, www.silicongenetics.com, 2005.

Google Scholar

[30]

L. Heath and N. Ramakrishnan, “The Emerging Landscape of Bioinformatics Software Systems,” Computer, vol. 35, no. 7, pp. 41-45, July 2002.

Digital Library

Google Scholar

[31]

G. Churchill, “Fundamentals of Experimental Design for cDNA Microarrays,” Nature Genetics, vol. 32, pp. 490-495, 2002.

Crossref

Google Scholar

[32]

J. Quackenbush, “Microarray Data Normalization and Transformation,” Nature Genetics, vol. 32, pp. 496-501, 2002.

Crossref

Google Scholar

[33]

K.A. Duca H. Goto Y. Kawaoka and J. Yin, “Time-Resolved mRNA Profiling during Influenza Infection: Extracting Information from a Challenging Experimental System,” Am. Soc. Virology, 20th Ann. Meeting, http://infovis.cs.vt.edu/cs5764/ fall2003/ideas/influenza.doc, 2001.

Google Scholar

[34]

G. Geiss M. Salvatore T. Tumpey V. Carter X. Wang C. Basler J. Taubenberger R. Bumbarner P. Palese M. Katze and A. Garcia-Sastre, “Cellular Transcriptional Profiling in Influenza A Virus-Infected Lung Epithelial Cells: The Role of the Nonstructural NS1 Protein in the Evasion of the Host Innate Defense and Its Potential Contribution to Pandemic Influenza,” PNAS, vol. 99, no. 16, pp.nbsp10736-10741, 2002.

Crossref

Google Scholar

[35]

E. Baechler F Batliwala G. Karvpis P. Gaffney W. Ortmann K. Espe K. Shark W. Grande K. Hughes K. Kapur P. Gregersen and T. Behrens, “Interferon-Inducible Gene Expression Signature in Peripheral Blood Cells of Patients with Severe SLE,” Proc. Nat'l Academy of Science, vol. 100, no. 5, pp. 2610-2615, 2003.

Crossref

Google Scholar

[36]

Affymetrix Technologies, http://www.affymetrix.com/, 2005.

Google Scholar

[37]

M. Eisen P Shellman P Brown and D Bostein, “Cluster Analysis and Display of Genome-Wide Expression Patterns,” Proc. Nat'l Academy of Science, vol. 95, no. 25, pp. 14963-14968, 1998.

Crossref

Google Scholar

[38]

H. Hochheiser E.H. Baehrecke S.M. Mount and B. Shneiderman, “Dynamic Querying for Pattern Identification in Microarray and Genomic Data,” Proc. IEEE Int'l Conf. Multimedia and Expo, 2003.

Digital Library

Google Scholar

[39]

J. Seo and B. Shneiderman, “Interactively Exploring Hierarchical Clustering Results,” Computer, vol. 35, pp. 80-86, 2002.

Digital Library

Google Scholar

[40]

J. Seo and B. Shneiderman, “A Rank-by-Feature Framework for Unsupervised Multidimensional Data Exploration Using Low Dimensional Projections,” Proc. IEEE InfoVis, 2004.

Digital Library

Google Scholar

[41]

Biocarta(TM), Charting Pathways of Life, http://www.biocarta.com/genes/index.asp, 2005.

Google Scholar

[42]

P. Saraiya C. North and K. Duca, “Visualization for Biological Pathways: Requirements Analysis, Systems Evaluation and Research Agenda,” Information Visualization, vol. 4, no. 3, 2005.

Digital Library

Google Scholar

Cited By

View all

Quadri GWang AWang ZAdorno JRosen PSzafir D(2024)Do You See What I See? A Qualitative Study Eliciting High-Level Visualization ComprehensionProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642813(1-26)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642813
Rezaie MTory MCarpendale S(2024)Struggles and Strategies in Understanding Information VisualizationsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.338856030:6(3035-3048)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1109/TVCG.2024.3388560
Nowak SBartram L(2024)Designing for Ambiguity in Visual Analytics: Lessons from Risk Assessment and PredictionIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.332657130:1(924-933)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TVCG.2023.3326571
Show More Cited By

Index Terms

An Insight-Based Methodology for Evaluating Bioinformatics Visualizations

Recommendations

An Insight-Based Longitudinal Study of Visual Analytics

Visualization tools are typically evaluated in controlled studies that observe the short-term usage of these tools by participants on preselected data sets and benchmark tasks. Though such studies provide useful suggestions, they miss the long-term ...
Constructing Interactive Visualizations with iVoLVER
CHI EA '16: Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems

iVoLVER, the Interactive Visual Language for Visualization Extraction and Reconstruction, is a web-based pen and touch system that graphically supports the construction of interactive visualizations and allows the extraction of data from different types ...
Multiscale Time Activity Data Exploration via Temporal Clustering Visualization Spreadsheet

Time-varying data is usually explored by animation or arrays of static images. Neither is particularly effective for classifying data by different temporal activities. Important temporal trends can be missed due to the lack of ability to find them with ...

Reviews

Reviewer: Bernice T. Glenn

The authors, whose experience is in bioinformatics and information visualization, work with information that results in very large data sets. Their paper is based on the premise that visualization's key purpose is to gain insight from the data that, in turn, could spark discovery. They consider that visualizations, based on their presentation form, should also have the ability to elicit questions that can lead to new hypotheses, and that seeing something that had passed unnoticed earlier can lead to new discoveries. The authors assert that their method for visualization analysis can also be applied to other domains. Previous visualization evaluations by others have typically focused on predetermined tasks and controlled measurements. Here, the authors propose a procedure that focuses on insights gained by actual explorations of the visuals as presented by a dataset. The questions they asked were: How do various visualization techniques affect how a user perceives the data__?__ Which tool generates the most relevant insight__?__ How successful is the tool in affecting the user's perception of the data__?__ How do visualizations support hypothesis generation__?__ What is the ability of the visualization for suggesting directions for future evaluation__?__ The experiment covered three multidimensional microarray data sets. The visualizations for each tool are described, covering scatter plots, histograms, bar charts, and manipulation of the data through zooming, changing the color ranges, and so on. The following visualization tools were used: Clusterview, TimeSearcher, HCE, Spotfire, and GeneSpring. Thirty test subjects were given a list of questions for each data set, and each participant was given a tool that he or she had not used before. The authors evaluated the average time to first insight for each tool, the amount learned by a participant using a particular tool, and the directed versus unexpected insights that were achieved with each tool for each data set. In this excellently designed study, the authors conclude that their experiment shows promise in measuring insight based on data visualization, but that the work was labor intensive and required domain expertise and motivated participants. They also found that the software design of interaction techniques played an important role in determining the effectiveness of the visualization.

Access critical reviews of Computing literature here

Become a reviewer for Computing Reviews.

Comments

Information & Contributors

Information

Published In

cover image IEEE Transactions on Visualization and Computer Graphics

IEEE Transactions on Visualization and Computer Graphics Volume 11, Issue 4

July 2005

127 pages

ISSN:1077-2626

Issue’s Table of Contents

Publisher

IEEE Educational Activities Department

United States

Publication History

Published: 01 July 2005

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

120
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 03 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Quadri GWang AWang ZAdorno JRosen PSzafir D(2024)Do You See What I See? A Qualitative Study Eliciting High-Level Visualization ComprehensionProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642813(1-26)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642813
Rezaie MTory MCarpendale S(2024)Struggles and Strategies in Understanding Information VisualizationsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.338856030:6(3035-3048)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1109/TVCG.2024.3388560
Nowak SBartram L(2024)Designing for Ambiguity in Visual Analytics: Lessons from Risk Assessment and PredictionIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.332657130:1(924-933)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TVCG.2023.3326571
Filipov VArleo ABögl MMiksch S(2024)On Network Structural and Temporal Encodings: A Space and Time OdysseyIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.331001930:8(5847-5860)Online publication date: 1-Aug-2024
https://dl.acm.org/doi/10.1109/TVCG.2023.3310019
Wambecke JLogre IMurgue A(2023)Catalogue Visu: a Tool for Fast Visualization PrototypingProceedings of the 34th Conference on l'Interaction Humain-Machine10.1145/3583961.3583969(1-10)Online publication date: 3-Apr-2023
https://dl.acm.org/doi/10.1145/3583961.3583969
van der Linden SSevastjanova RFunk MEl-Assady M(2023)MediCoSpace: Visual Decision-Support for Doctor-Patient Consultations using Medical Concept Spaces from EHRsACM Transactions on Management Information Systems10.1145/356427514:2(1-20)Online publication date: 25-Jan-2023
https://dl.acm.org/doi/10.1145/3564275
Akbaba DLange DCorrell MLex AMeyer M(2023)Troubling Collaboration: Matters of Care for Visualization Design StudyProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581168(1-15)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581168
Koonchanok RTawde GNarayanasamy GWalimbe SReda K(2023)Visual Belief Elicitation Reduces the Incidence of False DiscoveryProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580808(1-17)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3580808
Battle LOttley A(2023)What Do We Mean When We Say “Insight”? A Formal Synthesis of Existing TheoryIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.332669830:9(6075-6088)Online publication date: 24-Oct-2023
https://dl.acm.org/doi/10.1109/TVCG.2023.3326698
Neumayr TAugstein MKubicek B(2022)Territoriality in Hybrid CollaborationProceedings of the ACM on Human-Computer Interaction10.1145/35552246:CSCW2(1-37)Online publication date: 11-Nov-2022
https://dl.acm.org/doi/10.1145/3555224
Show More Cited By

Abstract

References

Cited By

Index Terms

Recommendations

An Insight-Based Longitudinal Study of Visual Analytics

Constructing Interactive Visualizations with iVoLVER

Multiscale Time Activity Data Exploration via Temporal Clustering Visualization Spreadsheet

Reviews

Access critical reviews of Computing literature here

Comments

Information

Published In

Publisher

Publication History

Author Tags

Qualifiers

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

View options

Figures

Other

Share

Share this Publication link

Share on social media

Affiliations