research-article

EnsembleMatrix: interactive visualization to support machine learning with multiple classifiers

Authors:

Desney S. TanAuthors Info & Claims

CHI '09: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

Pages 1283 - 1292

https://doi.org/10.1145/1518701.1518895

Published: 04 April 2009 Publication History

Abstract

Machine learning is an increasingly used computational tool within human-computer interaction research. While most researchers currently utilize an iterative approach to refining classifier models and performance, we propose that ensemble classification techniques may be a viable and even preferable alternative. In ensemble learning, algorithms combine multiple classifiers to build one that is superior to its components. In this paper, we present EnsembleMatrix, an interactive visualization system that presents a graphical view of confusion matrices to help users understand relative merits of various classifiers. EnsembleMatrix allows users to directly interact with the visualizations in order to explore and build combination models. We evaluate the efficacy of the system and the approach in a user study. Results show that users are able to quickly combine multiple classifiers operating on multiple feature sets to produce an ensemble classifier with accuracy that approaches best-reported performance classifying images in the CalTech-101 dataset.

References

[1]

Ankerst, M., Elsen, C., Ester, M. and Kriegel, H.-P. Visual classification: an interactive approach to decision tree construction. Proc. KDD 1999, (1999), 392--396.

Digital Library

[2]

Becker, B., Kohavi, R. and Sommerfield, D. Visualizing the Simple Bayesian Classifier. {ed.} Fayyad, U., Grinstein, G. and Wierse, A. (2001), 237--249.

[3]

Bertin, J. Semiology of graphics. University of Wisconsin Press, Madison, WI, USA,1984.

Digital Library

[4]

Bosch, A., Zisserman, A. and Munoz, X. Representing shape with a spatial pyramid kernel. Proc. CIVR 2007, (2007), 401--408.

Digital Library

[5]

Breiman, L. Bagging predictors. Machine Learning (1996), 123--140.

Digital Library

[6]

Caragea, D., Cook, D. and Honavar, V.G. Gaining insights into support vector machine pattern classifiers using projection-based tour methods. Proc. KDD 2001, (2001), 251--256.

Digital Library

[7]

Dai, J. and Cheng, J. HMMEditor: a visual editing tool for profile hidden Markov models. BMC Genomics 2008, 9 (2008).

[8]

Drummond, C. and Holte, R.C. Cost curves: An improved method for visualizing classifier performance. Machine Learning, 65, 1 (2006), 95--130.

Digital Library

[9]

Evgeniou, T., Pontil, M. and Elisseeff, A. Leave One Out Error, Stability, and Generalization of Voting Combinations of Classifiers. Machine Learning, 55 (2004), 71--97.

Digital Library

[10]

Fails, J.A. and Olsen, D.R.J. Interactive machine learning. Proc. IUI 2003, (2003), 39--45.

Digital Library

[11]

Fei-Fei, L., Fergus, R. and Perona, P. Learning Generative Visual Models from Few Training Examples: An Incremental Bayesian Approach Tested on 101 Object Categories. CVPRW, 12 (2004), 178.

Digital Library

[12]

Fogarty, J., Ko, A.J., Aung, H.H., Golden, E., Tang, K.P. and Hudson, S.E. Examining task engagement in sensor-based statistical models of human interruptibility. Proc. CHI 2005, (2005), 331--340.

Digital Library

[13]

Frank, E. and Hall, M. Visualizing class probability estimators. Lecture Notes in Artificial Intelligence 2838, Springer (2003), 168--179.

[14]

Garner, S.R. WEKA: The Waikato Environment for Knowledge Analysis. Proc. New Zealand Computer Science Research Students Conference (1995), 57--64.

[15]

Grauman, K. and Darrell, T. The Pyramid Match Kernel: Discriminative Classification with Sets of Image Features. Proc. ICCV 2005, (2005), 1458--1465.

Digital Library

[16]

Grimes, D., Tan, D.S., Hudson, S.E., Shenoy, P. and Rao, R.P. Feasibility and pragmatics of classifying working memory load with an electroencephalograph. Proc. CHI 2008, (2008), 835--844.

Digital Library

[17]

Henry, N. and Fekete, J.-D. MatrixExplorer: a Dual-Representation System to Explore Social Networks. IEEE Trans Visualization and Computer Graphics, 12, (2006), 677--684.

Digital Library

[18]

Kapoor, A., Grauman, K., Urtasun, R. and Darrell, T. Active Learning with Gaussian Processes for Object Categorization. Proc. ICCV 2007, (2007), 1--8.

[19]

Kapoor, A. and Horvitz, E. Experience sampling for building predictive user models: a comparative study. Proc. CHI 2008, (2008), 657--666.

Digital Library

[20]

Kim, B., Lee, B. and Seo, J. Visualizing set concordance with permutation matrices and fan diagrams. Interacting with Computers, 19, 5-6 (2007), 630--643.

Digital Library

[21]

Kittler, J., Hatef, M., Duin, R.P. and Matas, J. On Combining Classifiers. IEEE TPAMI, 20, 3 (1998), 226--239.

Digital Library

[22]

Lazebnik, S., Schmid, C. and Ponce, J. Beyond Bags of Features: Spatial Pyramid Matching for Recognizing Natural Scene Categories. Proc. CVPR 2006, (2006), 2169--2178.

Digital Library

[23]

Mäkinen, E. and Siirtola, H. The Barycenter Heuristic and the Reorderable Matrix. Informatica (Slovenia), 29, 3 (2005), 357--364.

[24]

Meyer, M., Lee, H., Barr, A. and Desbrun, M. Generalized Barycentric Coordinates on Irregular Polygons. Journal of Graphics Tools, 7, 1 (2002), 13--22.

Digital Library

[25]

Moustakis, V. Do People in HCI Use Machine Learning? HCI, 2, (1997), 95--98.

[26]

Oliver, N., Garg, A. and Horvitz, E. Layered representations for learning and inferring office activity from multiple sensory channels. Computer Vision and Image Understanding, 96, 2 (2004) 163--180.

Digital Library

[27]

Patel, K., Fogarty, J., Landay, J.A. and Harrison, B. Examining Difficulties Software Developers Encounter in the Adoption of Statistical Machine Learning. Proc. AAAI 2008, (2008), 1563--1566.

Digital Library

[28]

Picard, R.W. Affective computing. MIT Press, Cambridge, MA, USA, 1997.

Digital Library

[29]

Rheingans, P. and desJardins, M. Visualizing High-Dimensional Predictive Model Quality. Proc. VIS 2000, (2000), 493--496.

Digital Library

[30]

Saponas, T.S., Tan, D.S., Morris, D. and Balakrishnan, R. Demonstrating the feasibility of using forearm electromyography for muscle-computer interfaces. Proc. CHI 2008, (2008), 515--524.

Digital Library

[31]

Schapire, R.E. The boosting approach to machine learning: An overview. Nonlinear Estimation and Classification. Springer (2003).

[32]

Stiglic, G., Mertik, M., Podgorelec, V. and Kokol, P. Using Visual Interpretation of Small Ensembles in Microarray Analysis. Proc. CMBS 2006, (2006), 691--695.

Digital Library

[33]

Urbanek, S. Exploring Statistical Forests. Proc. of the 2002 Joint Statistical Meeting, Springer (2002).

[34]

Varma, M. and Ray, D. Learning The Discriminative Power-Invariance Trade-Off. Proc. ICCV 2007, (2007), 1--8.

[35]

Wang, J., Yu, B. and Gasser, L. Classification Visualization with Shaded Similarity Matrix, Technical Report, GSLIS University of Illinois at Urbana-Champaign, (2002).

[36]

Ware, M., Frank, E., Holmes, G., Hall, M. and Witten, I.H. Interactive machine learning: letting users build classifiers. Int. J. Human-Computer Studies, 55, 3 (2001), 281--292.

Digital Library

[37]

Zhang, H., Berg, A. C., Maire, M. and Malik, J. SVM-KNN: Discriminative Nearest Neighbor Classification for Visual Category Recognition. Proc. CVPR 2006, (2006), 2126--2136.

Digital Library

Cited By

Doppalapudi BRahman MRosen P(2025)Seeing Is Believing: The Role of Scatterplots in Recommender System Trust and Decision-MakingAdvances in Visual Computing10.1007/978-3-031-77392-1_32(425-438)Online publication date: 22-Jan-2025
https://doi.org/10.1007/978-3-031-77392-1_32
Arrubla-Hoyos WCarrascal-Porras FGómez J(2024)Predicción de riesgo cardiovascular mediante aprendizaje automático: un análisis comparativo entre técnicasIngeniería y Competitividad10.25100/iyc.v26i1.1322926:1Online publication date: 26-Feb-2024
https://doi.org/10.25100/iyc.v26i1.13229
Kawabe WSugano Y(2024)Image-to-Text Translation for Interactive Image Recognition: A Comparative User Study with Non-expert UsersJournal of Information Processing10.2197/ipsjjip.32.35832(358-368)Online publication date: 2024
https://doi.org/10.2197/ipsjjip.32.358
Show More Cited By

Index Terms

EnsembleMatrix: interactive visualization to support machine learning with multiple classifiers
1. Computing methodologies
  1. Machine learning
2. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction paradigms
      1. Graphical user interfaces

Recommendations

Interactive optimization for steering machine classification
CHI '10: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

Interest has been growing within HCI on the use of machine learning and reasoning in applications to classify such hidden states as user intentions, based on observations. HCI researchers with these interests typically have little expertise in machine ...
Optimizing clustering to promote data diversity when generating an ensemble classifier
GECCO '18: Proceedings of the Genetic and Evolutionary Computation Conference Companion

In this paper, we propose a method to generate an optimized ensemble classifier. In the proposed method, a diverse input space is created by clustering training data incrementally within a cycle. A cycle is one complete round that includes clustering, ...
Comparison of ensemble learning methods applied to network intrusion detection
ICC '17: Proceedings of the Second International Conference on Internet of things, Data and Cloud Computing

This paper investigates the possibility of using ensemble learning methods to improve the performance of intrusion detection systems. We compare an ensemble of three ensemble learning methods, boosting, bagging and stacking in order to improve the ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '09: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

April 2009

2426 pages

ISBN:9781605582467

DOI:10.1145/1518701

General Chairs:
Dan R. Olsen
Brigham Young University
,
Richard B. Arthur
Brigham Young University
,
Program Chairs:
Ken Hinckley
Microsoft Research
,
Meredith Ringel Morris
Microsoft Research
,
Scott Hudson
Carnegie Mellon University
,
Saul Greenberg
University of Calgary

Copyright © 2009 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 04 April 2009

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

CHI '09

Sponsor:

CHI '09: CHI Conference on Human Factors in Computing Systems

April 4 - 9, 2009

MA, Boston, USA

Acceptance Rates

CHI '09 Paper Acceptance Rate 277 of 1,130 submissions, 25%;

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI 2025

Sponsor:
sigchi

ACM CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

178
Total Citations
View Citations
2,153
Total Downloads

Downloads (Last 12 months)116
Downloads (Last 6 weeks)12

Reflects downloads up to 20 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Doppalapudi BRahman MRosen P(2025)Seeing Is Believing: The Role of Scatterplots in Recommender System Trust and Decision-MakingAdvances in Visual Computing10.1007/978-3-031-77392-1_32(425-438)Online publication date: 22-Jan-2025
https://doi.org/10.1007/978-3-031-77392-1_32
Arrubla-Hoyos WCarrascal-Porras FGómez J(2024)Predicción de riesgo cardiovascular mediante aprendizaje automático: un análisis comparativo entre técnicasIngeniería y Competitividad10.25100/iyc.v26i1.1322926:1Online publication date: 26-Feb-2024
https://doi.org/10.25100/iyc.v26i1.13229
Kawabe WSugano Y(2024)Image-to-Text Translation for Interactive Image Recognition: A Comparative User Study with Non-expert UsersJournal of Information Processing10.2197/ipsjjip.32.35832(358-368)Online publication date: 2024
https://doi.org/10.2197/ipsjjip.32.358
Fujita KUwaseki KBu HTakashima KKitamura Y(2024)ConfusionLens: Focus+Context Visualization Interface for Performance Analysis of Multiclass Image ClassifiersACM Transactions on Interactive Intelligent Systems10.1145/370013915:1(1-21)Online publication date: 17-Oct-2024
https://dl.acm.org/doi/10.1145/3700139
Wei JXia DXie HChang CLi CYang X(2024)SpaceEditing: A Latent Space Editing Interface for Integrating Human Knowledge into Deep Neural NetworksProceedings of the 29th International Conference on Intelligent User Interfaces10.1145/3640543.3645211(489-503)Online publication date: 18-Mar-2024
https://dl.acm.org/doi/10.1145/3640543.3645211
Thomas IOh SSzafir D(2024)Assessing User Trust in Active Learning Systems: Insights from Query Policy and Uncertainty VisualizationProceedings of the 29th International Conference on Intelligent User Interfaces10.1145/3640543.3645207(772-786)Online publication date: 18-Mar-2024
https://dl.acm.org/doi/10.1145/3640543.3645207
Kim YLee ELee YOh U(2024)Understanding Novice's Annotation Process For 3D Semantic Segmentation Task With Human-In-The-LoopProceedings of the 29th International Conference on Intelligent User Interfaces10.1145/3640543.3645150(444-454)Online publication date: 18-Mar-2024
https://dl.acm.org/doi/10.1145/3640543.3645150
Bhattacharya AStumpf SGosak LStiglic GVerbert K(2024)EXMOS: Explanatory Model Steering through Multifaceted Explanations and Data ConfigurationsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642106(1-27)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642106
Yu ZNamkung HGuo JMilner HGoldfoot JWang YSekar V(2024)SEAM-EZ: Simplifying Stateful Analytics through Visual ProgrammingProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642055(1-23)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642055
Lei FMa YFotheringham AMack ELi ZSachdeva MBardin SMaciejewski R(2024)GeoExplainer: A Visual Analytics Framework for Spatial Modeling Contextualization and Report GenerationIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.332735930:1(1391-1401)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TVCG.2023.3327359
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents