research-article

AnchorViz: Facilitating Classifier Error Discovery through Interactive Semantic Data Exploration

Authors:

Steven Drucker,

Patrice SimardAuthors Info & Claims

IUI '18: Proceedings of the 23rd International Conference on Intelligent User Interfaces

Pages 269 - 280

https://doi.org/10.1145/3172944.3172950

Published: 05 March 2018 Publication History

Abstract

When building a classifier in interactive machine learning, human knowledge about the target class can be a powerful reference to make the classifier robust to unseen items. The main challenge lies in finding unlabeled items that can either help discover or refine concepts for which the current classifier has no corresponding features (i.e., it has feature blindness). Yet it is unrealistic to ask humans to come up with an exhaustive list of items, especially for rare concepts that are hard to recall. This paper presents AnchorViz, an interactive visualization that facilitates error discovery through semantic data exploration. By creating example-based anchors, users create a topology to spread data based on their similarity to the anchors and examine the inconsistencies between data points that are semantically related. The results from our user study show that AnchorViz helps users discover more prediction errors than stratified random and uncertainty sampling methods.

References

[1]

Jae-wook Ahn and Peter Brusilovsky. 2009. Adaptive visualization of search results: Bringing user models to visual analytics. Information Visualization 8, 3 (2009), 167--179.

Digital Library

[2]

Saleema Amershi, Maya Cakmak, William Bradley Knox, and Todd Kulesza. 2014. Power to the people: The role of humans in interactive machine learning. AI Magazine 35, 4 (2014), 105--120.

Digital Library

[3]

Joshua Attenberg, Panos Ipeirotis, and Foster Provost. 2015. Beat the Machine: Challenging Humans to Find a Predictive Model's “Unknown Unknowns”. Journal of Data and Information Quality (JDIQ) 6, 1 (2015), 1.

Digital Library

[4]

Josh Attenberg and Foster Provost. 2010. Why label when you can search?: alternatives to active learning for applying human resources to build classification models under extreme class imbalance. In Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, 423--432.

Digital Library

[5]

Allan M Collins and M Ross Quillian. 1969. Retrieval time from semantic memory. Journal of verbal learning and verbal behavior 8, 2 (1969), 240--247.

[6]

Inderjit S Dhillon and Dharmendra S Modha. 2001. Concept decompositions for large sparse text data using clustering. Machine learning 42, 1 (2001), 143--175.

[7]

Inderjit Singh Dhillon and Dharmendra Shantilal Modha. 2003. Concept decomposition using clustering. (May 6 2003). US Patent 6,560,597.

[8]

Jasminka Dobsa and BJ Basic. 2003. Concept decomposition by fuzzy k-means algorithm. In Web Intelligence, 2003. WI 2003. Proceedings. IEEE/WIC International Conference on. IEEE, 684--688.

Digital Library

[9]

Jerry Alan Fails and Dan R. Olsen, Jr. 2003. Interactive Machine Learning. In Proceedings of the 8th International Conference on Intelligent User Interfaces (IUI '03). ACM, New York, NY, USA, 39--45.

Digital Library

[10]

Patrick E Hoffman. Table visualizations: a formal model and its applications. Ph.D. Dissertation. University of Massachusetts. Lowell.

Digital Library

[11]

Camille Jandot, Patrice Simard, Max Chickering, David Grangier, and Jina Suh. 2016. Interactive Semantic Featuring for Text Classification. arXiv preprint arXiv:1606.07545 (2016).

[12]

Todd Kulesza, Saleema Amershi, Rich Caruana, Danyel Fisher, and Denis Charles. 2014. Structured labeling for facilitating concept evolution in machine learning. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 3075--3084.

Digital Library

[13]

Himabindu Lakkaraju, Ece Kamar, Rich Caruana, and Eric Horvitz. 2017. Identifying Unknown Unknowns in the Open World: Representations and Policies for Guided Exploration. In AAAI. 2124--2132.

[14]

Hanseung Lee, Jaeyeon Kihm, Jaegul Choo, John Stasko, and Haesun Park. 2012. iVisClustering: An Interactive Visual Document Clustering via Topic Modeling. Computer Graphics Forum 31, 3pt3 (June 2012), 1155--1164.0.1145/1518701.1518895 00097.

Digital Library

[15]

Endel Tulving and others. 1972. Episodic and semantic memory. Organization of memory 1 (1972), 381--403.

[16]

Malcolm Ware, Eibe Frank, Geoffrey Holmes, Mark Hall, and Ian H Witten. 2001. Interactive machine learning: letting users build classifiers. International Journal of Human-Computer Studies 55, 3 (2001), 281--292.

Digital Library

Cited By

Wei JXia DXie HChang CLi CYang X(2024)SpaceEditing: A Latent Space Editing Interface for Integrating Human Knowledge into Deep Neural NetworksProceedings of the 29th International Conference on Intelligent User Interfaces10.1145/3640543.3645211(489-503)Online publication date: 18-Mar-2024
https://dl.acm.org/doi/10.1145/3640543.3645211
Wei JChang CYang XIgarashi T(2024)CanvasPic: An Interactive Tool for Freely Generating Facial Images Based on Spatial LayoutExtended Abstracts of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613905.3650952(1-8)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613905.3650952
Lam MTeoh JLanday JHeer JBernstein M(2024)Concept Induction: Analyzing Unstructured Text with High-Level Concepts Using LLooMProceedings of the CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642830(1-28)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642830
Show More Cited By

Index Terms

AnchorViz: Facilitating Classifier Error Discovery through Interactive Semantic Data Exploration

Recommendations

AnchorViz: Facilitating Semantic Data Exploration and Concept Discovery for Interactive Machine Learning
Special Issue on IUI 2018

When building a classifier in interactive machine learning (iML), human knowledge about the target class can be a powerful reference to make the classifier robust to unseen items. The main challenge lies in finding unlabeled items that can either help ...
Transductive Multilabel Learning via Label Set Propagation

The problem of multilabel classification has attracted great interest in the last decade, where each instance can be assigned with a set of multiple class labels simultaneously. It has a wide variety of real-world applications, e.g., automatic image ...
EnsembleMatrix: interactive visualization to support machine learning with multiple classifiers
CHI '09: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

Machine learning is an increasingly used computational tool within human-computer interaction research. While most researchers currently utilize an iterative approach to refining classifier models and performance, we propose that ensemble classification ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

IUI '18: Proceedings of the 23rd International Conference on Intelligent User Interfaces

March 2018

698 pages

ISBN:9781450349451

DOI:10.1145/3172944

General Chairs:
Shlomo Berkovsky
CSIRO, Australia
,
Yoshinori Hijikata
Kwansei Gakuin University, Japan
,
Jun Rekimoto
University of Tokyo, Japan
,
Program Chairs:
Margaret Burnett
Oregon State University, USA
,
Mark Billinghurst
University of South Australia, Australia
,
Aaron Quigley
University of St Andrews, UK

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGAI: ACM Special Interest Group on Artificial Intelligence

In-Cooperation

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 05 March 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

IUI'18

Sponsor:

SIGAI

IUI'18: 23rd International Conference on Intelligent User Interfaces

March 7 - 11, 2018

Tokyo, Japan

Acceptance Rates

IUI '18 Paper Acceptance Rate 43 of 299 submissions, 14%;

Overall Acceptance Rate 746 of 2,811 submissions, 27%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

30
Total Citations
View Citations
514
Total Downloads

Downloads (Last 12 months)19
Downloads (Last 6 weeks)1

Reflects downloads up to 26 Jul 2024

Other Metrics

View Author Metrics

Citations

Cited By

Wei JXia DXie HChang CLi CYang X(2024)SpaceEditing: A Latent Space Editing Interface for Integrating Human Knowledge into Deep Neural NetworksProceedings of the 29th International Conference on Intelligent User Interfaces10.1145/3640543.3645211(489-503)Online publication date: 18-Mar-2024
https://dl.acm.org/doi/10.1145/3640543.3645211
Wei JChang CYang XIgarashi T(2024)CanvasPic: An Interactive Tool for Freely Generating Facial Images Based on Spatial LayoutExtended Abstracts of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613905.3650952(1-8)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613905.3650952
Lam MTeoh JLanday JHeer JBernstein M(2024)Concept Induction: Analyzing Unstructured Text with High-Level Concepts Using LLooMProceedings of the CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642830(1-28)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642830
Gómez-Carmona OCasado-Mansilla DLópez-de-Ipiña DGarcía-Zubia J(2024)Human-in-the-loop machine learning: Reconceptualizing the role of the user in interactive approachesInternet of Things10.1016/j.iot.2023.10104825(101048)Online publication date: May-2024
https://doi.org/10.1016/j.iot.2023.101048
Rastogi CTulio Ribeiro MKing NNori HAmershi S(2023)Supporting Human-AI Collaboration in Auditing LLMs with LLMsProceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society10.1145/3600211.3604712(913-926)Online publication date: 8-Aug-2023
https://dl.acm.org/doi/10.1145/3600211.3604712
Cabrera ÁPerer AHong J(2023)Improving Human-AI Collaboration With Descriptions of AI BehaviorProceedings of the ACM on Human-Computer Interaction10.1145/35796127:CSCW1(1-21)Online publication date: 16-Apr-2023
https://dl.acm.org/doi/10.1145/3579612
Cabrera ÁFu EBertucci DHolstein KTalwalkar AHong JPerer A(2023)Zeno: An Interactive Framework for Behavioral Evaluation of Machine LearningProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581268(1-14)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581268
Cabrera ÁTulio Ribeiro MLee BDeline RPerer ADrucker S(2023)What Did My AI Learn? How Data Scientists Make Sense of Model BehaviorACM Transactions on Computer-Human Interaction10.1145/354292130:1(1-27)Online publication date: 7-Mar-2023
https://dl.acm.org/doi/10.1145/3542921
Bäuerle ACabrera ÁHohman FMaher MKoski DSuau XBarik TMoritz D(2022)Symphony: Composing Interactive Interfaces for Machine LearningProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3502102(1-14)Online publication date: 29-Apr-2022
https://dl.acm.org/doi/10.1145/3491102.3502102
Dodge JAnderson AOlson MDikkala RBurnett M(2022)How Do People Rank Multiple Mutant Agents?Proceedings of the 27th International Conference on Intelligent User Interfaces10.1145/3490099.3511115(191-211)Online publication date: 22-Mar-2022
https://dl.acm.org/doi/10.1145/3490099.3511115
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents