research-article

ModelTracker: Redesigning Performance Analysis Tools for Machine Learning

Authors:

Saleema Amershi,

Max Chickering,

Steven M. Drucker,

Patrice Simard,

Jina SuhAuthors Info & Claims

CHI '15: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems

Pages 337 - 346

https://doi.org/10.1145/2702123.2702509

Published: 18 April 2015 Publication History

Abstract

Model building in machine learning is an iterative process. The performance analysis and debugging step typically involves a disruptive cognitive switch from model building to error analysis, discouraging an informed approach to model building. We present ModelTracker, an interactive visualization that subsumes information contained in numerous traditional summary statistics and graphs while displaying example-level performance and enabling direct error examination and debugging. Usage analysis from machine learning practitioners building real models with ModelTracker over six months shows ModelTracker is used often and throughout model building. A controlled experiment focusing on ModelTracker's debugging capabilities shows participants prefer ModelTracker over traditional tools without a loss in model performance.

References

[1]

Ankerst, M., Elsen, C., Ester, M., and Kriegal, H. Visual Classification: An Interactive Approach to Decision Tree Construction. Proc. KDD 1999, ACM Press (1999), 392--396.

Digital Library

[2]

Becker, B., Kohavi, R., and Sommerfield, D. Visualizing the Simple Bayesian Classifier. Information Visualization in Data Mining and Knowledge Discovery. Fayyad, U., Grinstein, G.G., and Wierse, A. (eds). Morgan Kaufmann Publishers, 2001, 237--249.

Digital Library

[3]

Bird, S., Klein, E., and Loper, E. Natural Language Processing with Python. O'Reilly Media, 2009.

Digital Library

[4]

Broekens, J., Cocx, T., and Kosters, W. Object-Centered Interactive Multi-Dimensional Scaling: Ask the Expert. Proc. BNAIC 2006, 59--66.

[5]

Caragea, D., Cook, D., and Honavar, V. Gaining Insights into Support Vector Machine Pattern Classifiers Using Projection-Based Tour Methods. Proc. KDD 2001, ACM Press (2001), 251--256.

Digital Library

[6]

Chan, Y., Correa, C., and Ma, K-L. Flow-based Scatterplots for Sensitivity Analysis. Proc. VAST 2010, IEEE (2010), 43--50.

[7]

Choo, J., Hanseung, L., Liu, Z., Stasko, J., and Park, H. An Interactive Visual Testbed System for Dimension Reduction and Clustering of Large-Scale HighDimensional Data. Proc. SPIE Electronic Imaging 2013, 865402-865402-15.

[8]

Domingos, P. A Few Useful Things to Know about Machine Learning. CACM 55, 10 (2012), 78--87.

Digital Library

[9]

Fails, J.A. and Olsen, D.R. Interactive Machine Learning. Proc. IUI 2003, ACM Press (2003), 39--45.

Digital Library

[10]

Fiebrink, R., Cook, P.R., and Trueman, D. Human Model Evaluation in Interactive Supervised Learning. Proc. CHI 2011, ACM Press (2011), 147--156.

Digital Library

[11]

Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., and Witten, I.H. The WEKA Data Mining Software: An Update. SIGKDD Explorations 11, 1 (2009).

Digital Library

[12]

Hao, M.C., Dayal, U., Sharma, R.K., Keim, D.A., and Janetzko, H. Variable Binned Scatter Plots. Information Visualization 9, 3 (2010), 194--203.

Digital Library

[13]

MATLAB 9.0 and Statistics Toolbox Release 2014a, The MathWorks, Inc., Natick, Massachusetts, USA, http://www.mathworks.com/products/statistics, 2014.

[14]

Mayorga, A. and Gleicher, M. Scatterplots: Overcoming Overdraw in Scatter Plots. IEEE TVCG 19, 9 (2013), 1526--1538.

Digital Library

[15]

Nettleton, D. F., Orriols-Puig, A., and Fornells, A. A Study of the Effect of Different Types of Noise on the Precision of Supervised Learning Techniques. AI Review 33, 4 (2010), 275--306.

Digital Library

[16]

Patel, K., Bancroft, N., Drucker, S.M., Fogarty, J., Ko, A., and Landay, J.A. Gestalt: Integrated Support for Implementation and Analysis in Machine Learning Processes. Proc. UIST 2010, ACM Press (2010), 37--46.

Digital Library

[17]

Patel, K., Drucker, S.M., Fogarty, J., Kapoor, A., and Tan, D.S. Using Multiple Models to Understand Data Proc. IJCAI 2011, AAAI Press (2011), 1723--1728.

Digital Library

[18]

Patel, K., Fogarty, J., Landay, J.A., and Harrison, B. Examining Difficulties Software Developers Encounter in the Adoption of Statistical Machine Learning. Proc. AAAI 2008, AAAI Press (2008), 1563--1566.

Digital Library

[19]

R Core Team, "R: A Language and Environment for Statistical Computing," R Foundation for Statistical Computing, http://www.R-project.org, 2013.

[20]

Rossi, F. Visual Data Mining and Machine Learning Proc. ESANN 2006, 251--264.

[21]

Simard, P., Chickering, D., Lakshmiratan, A., Charles, D., Bottou, L., Suarez, C.G.J., Grangier, D., Amershi, S., Verwey, J., and Suh, J. ICE: Enabling Non-Experts to Build Models Interactively for Large-Scale Lopsided Problems. 2014, arXiv:1409.4814.

[22]

Talbot, J., Lee, B., Kapoor, A., and Tan, D. EnsembleMatrix: Interactive Visualization to Support Machine Learning with Multiple Classifiers. Proc. CHI 2009, ACM Press (2009), 1283--1292.

Digital Library

Cited By

Zhang HYan BCao LMadden SRundensteiner E(2024)MetaStore: Analyzing Deep Learning Meta-Data at ScaleProceedings of the VLDB Endowment10.14778/3648160.364818217:6(1446-1459)Online publication date: 1-Feb-2024
https://dl.acm.org/doi/10.14778/3648160.3648182
Ferdowsi MKwan BTan MSaedon NSubramaniam SAbu Hashim NMohd Nasir SZainal Abidin IChee KGoh C(2024)Classification of vasovagal syncope from physiological signals on tilt table testingBioMedical Engineering OnLine10.1186/s12938-024-01229-923:1Online publication date: 30-Mar-2024
https://doi.org/10.1186/s12938-024-01229-9
Wan CLiu SXie SLiu YHoffmann HMaire MLu S(2024)Keeper: Automated Testing and Fixing of Machine Learning SoftwareACM Transactions on Software Engineering and Methodology10.1145/3672451Online publication date: 13-Jun-2024
https://doi.org/10.1145/3672451
Show More Cited By

Index Terms

ModelTracker: Redesigning Performance Analysis Tools for Machine Learning
1. Human-centered computing
  1. Human computer interaction (HCI)

Recommendations

Heapviz: interactive heap visualization for program understanding and debugging
SOFTVIS '10: Proceedings of the 5th international symposium on Software visualization

Understanding the data structures in a program is crucial to understanding how the program works, or why it doesn't work. Inspecting the code that implements the data structures, however, is an arduous task and often fails to yield insights into the ...
Recent research advances on interactive machine learning

Interactive machine learning (IML) is an iterative learning process that tightly couples a human with a machine learner, which is widely used by researchers and practitioners to effectively solve a wide variety of real-world application problems. ...
PaintingClass: interactive construction, visualization and exploration of decision trees
KDD '03: Proceedings of the ninth ACM SIGKDD international conference on Knowledge discovery and data mining

Decision trees are commonly used for classification. We propose to use decision trees not just for classification but also for the wider purpose of knowledge discovery, because visualizing the decision tree can reveal much valuable information in the ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '15: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems

April 2015

4290 pages

ISBN:9781450331456

DOI:10.1145/2702123

General Chairs:
Bo Begole
Huawei, USA
,
Jinwoo Kim
Yonsei University, Korea
,
Program Chairs:
Kori Inkpen
Microsoft Research, USA
,
Woontack Woo
KAIST, Korea

Copyright © 2015 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 April 2015

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

CHI '15

Sponsor:

SIGCHI

CHI '15: CHI Conference on Human Factors in Computing Systems

April 18 - 23, 2015

Seoul, Republic of Korea

Acceptance Rates

CHI '15 Paper Acceptance Rate 486 of 2,120 submissions, 23%;

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

182
Total Citations
View Citations
1,884
Total Downloads

Downloads (Last 12 months)227
Downloads (Last 6 weeks)23

Reflects downloads up to 01 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Zhang HYan BCao LMadden SRundensteiner E(2024)MetaStore: Analyzing Deep Learning Meta-Data at ScaleProceedings of the VLDB Endowment10.14778/3648160.364818217:6(1446-1459)Online publication date: 1-Feb-2024
https://dl.acm.org/doi/10.14778/3648160.3648182
Ferdowsi MKwan BTan MSaedon NSubramaniam SAbu Hashim NMohd Nasir SZainal Abidin IChee KGoh C(2024)Classification of vasovagal syncope from physiological signals on tilt table testingBioMedical Engineering OnLine10.1186/s12938-024-01229-923:1Online publication date: 30-Mar-2024
https://doi.org/10.1186/s12938-024-01229-9
Wan CLiu SXie SLiu YHoffmann HMaire MLu S(2024)Keeper: Automated Testing and Fixing of Machine Learning SoftwareACM Transactions on Software Engineering and Methodology10.1145/3672451Online publication date: 13-Jun-2024
https://doi.org/10.1145/3672451
Ma WYang CKästner CBosch JLewis GCleland-Huang JMuccini H(2024)(Why) Is My Prompt Getting Worse? Rethinking Regression Testing for Evolving LLM APIsProceedings of the IEEE/ACM 3rd International Conference on AI Engineering - Software Engineering for AI10.1145/3644815.3644950(166-171)Online publication date: 14-Apr-2024
https://dl.acm.org/doi/10.1145/3644815.3644950
Shome ACruz LVan Deursen AIzadi MDi Sorbo APanichella S(2024)Towards Automatic Translation of Machine Learning Visual Insights to Analytical AssertionsProceedings of the Third ACM/IEEE International Workshop on NL-based Software Engineering10.1145/3643787.3648032(29-32)Online publication date: 20-Apr-2024
https://dl.acm.org/doi/10.1145/3643787.3648032
Hanafi MReiss FKatsis YMoore RWood DFalakmasir MLiu C(2024)Machine-Assisted Error Discovery in Conversational AI SystemsExtended Abstracts of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613905.3651120(1-10)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613905.3651120
Qian CReif EKahng M(2024)Understanding the Dataset Practitioners Behind Large Language ModelsExtended Abstracts of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613905.3651007(1-7)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613905.3651007
Kahng MTenney IPushkarna MLiu MWexler JReif EKallarackal KChang MTerry MDixon L(2024)LLM Comparator: Visual Analytics for Side-by-Side Evaluation of Large Language ModelsExtended Abstracts of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613905.3650755(1-7)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613905.3650755
Hohman FWang CLee JGörtler JMoritz DBigham JRen ZForet CShan QZhang X(2024)Talaria: Interactively Optimizing Machine Learning Models for Efficient InferenceProceedings of the CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642628(1-19)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642628
Hoque MMashiat TGhai BShelton CChevalier FKraus KElmqvist N(2024)The HaLLMark Effect: Supporting Provenance and Transparent Use of Large Language Models in Writing with Interactive VisualizationProceedings of the CHI Conference on Human Factors in Computing Systems10.1145/3613904.3641895(1-15)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3641895
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents