research-article

Improving User Confidence in Concept Maps: Exploring Data Driven Explanations

Authors:

Pierre Le Bras,

Thomas S. Methven,

Stefano Padilla,

Mike J. ChantlerAuthors Info & Claims

CHI '18: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems

Paper No.: 404, Pages 1 - 13

https://doi.org/10.1145/3173574.3173978

Published: 21 April 2018 Publication History

Abstract

Automated tools are increasingly being used to generate highly engaging concept maps as an aid to strategic planning and other decision-making tasks. Unless stakeholders can understand the principles of the underlying layout process, however, we have found that they lack confidence and are therefore reluctant to use these maps. In this paper, we present a qualitative study exploring the effect on users' confidence of using data-driven explanation mechanisms, by conducting in-depth scenario-based interviews with ten participants. To provide diversity in stimulus and approach we use two explanation mechanisms based on projection and agglomerative layout methods. The themes exposed in our results indicate that the data-driven explanations improved user confidence in several ways, and that process clarity and layout density also affected users' views of the credibility of the concept maps. We discuss how these factors can increase uptake of automated tools and affect user confidence.

Supplementary Material

suppl.mov (pn3429-file5.mp4)

Supplemental video

Download
5.07 MB

References

[1]

Gregory A Aarons, Rebecca S Wells, Karen Zagursky, Danielle L Fettes, and Lawrence A Palinkas. 2009. Implementing evidence-based practice in community mental health agencies: A multiple stakeholder analysis. American journal of public health 99, 11 (2009), 2087--2095.

[2]

Anne Adams and Anna L Cox. 2008. Questionnaires, in-depth interviews and focus groups. In Research Methods for Human-Computer Interaction, Paul Cairns and Anna L Cox (Eds.). Cambridge University Press, Chapter 2, 17--34.

[3]

Lynda A. Anderson, Margaret K. Gwaltney, Demia L. Sundra, Ross C. Brownson, Mary Kane, Alan W. Cross, Richard Mack, Randy Schwartz, Tom Sims, and Carol R. White. 2006. Using Concept Mapping to Develop a Logic Model for the Prevention Research Centers Program. Preventing Chronic Disease 3, 1 (2006).

[4]

Christine Barter and Emma Renold. 1999. The use of vignettes in qualitative research. Social research update 25, 9 (1999), 1--6.

[5]

Victoria Bellotti, Alexander Ambard, Daniel Turner, Christina Gossmann, Kamila Demkova, and John M Carroll. 2015. A muddle of models of motivation for using peer-to-peer economy systems. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, 1085--1094.

Digital Library

[6]

John Benson and Nicky Britten. 2002. Patients' decisions about whether or not to take antihypertensive drugs: qualitative study. Bmj 325, 7369 (2002), 873.

[7]

David M Blei. 2012. Probabilistic topic models. Commun. ACM 55, 4 (2012), 77--84.

Digital Library

[8]

David M Blei, Andrew Y Ng, and Michael I Jordan. 2003. Latent dirichlet allocation. Journal of machine Learning research 3, Jan (2003), 993--1022.

Digital Library

[9]

Michael Bostock, Vadim Ogievetsky, and Jeffrey Heer. 2011. D3 data-driven documents. IEEE transactions on visualization and computer graphics 17, 12 (2011), 2301--2309.

Digital Library

[10]

Kevin W Boyack, Richard Klavans, and Katy Börner. 2005. Mapping the backbone of science. Scientometrics 64, 3 (2005), 351--374.

[11]

Laila Burla, Birte Knierim, Jurgen Barth, Katharina Liewald, Margreet Duetz, and Thomas Abel. 2008. From text to codings: intercoder reliability assessment in qualitative content analysis. Nursing research 57, 2 (2008), 113--117.

[12]

John L Campbell, Charles Quincy, Jordan Osserman, and Ove K Pedersen. 2013. Coding in-depth semistructured interviews: Problems of unitization and intercoder reliability and agreement. Sociological Methods & Research 42, 3 (2013), 294--320.

[13]

Emilio Carrizosa, Vanesa Guerrero, and Dolores Romero Morales. 2017. Visualizing proportions and dissimilarities by space-filling maps: a large neighborhood search approach. Computers & Operations Research 78 (2017), 369--380.

Digital Library

[14]

Juliet Corbin and Anselm Strauss. 2008. Basics of qualitative research: techniques and procedures for developing grounded theory. 2008. (2008).

[15]

Dan Cosley, Shyong K Lam, Istvan Albert, Joseph A Konstan, and John Riedl. 2003. Is seeing believing?: how recommender system interfaces affect users' opinions. In Proceedings of the SIGCHI conference on Human factors in computing systems. ACM, 585--592.

Digital Library

[16]

Martin Davies. 2011. Concept mapping, mind mapping and argument mapping: what are the differences and do they matter? Higher education 62, 3 (2011), 279--301.

[17]

Niklas Elmqvist and Ji Soo Yi. 2015. Patterns for visualization evaluation. Information Visualization 14, 3 (2015), 250--269.

[18]

Michele Emmer. 2005. The visual mind II. (2005).

[19]

Holly J Falk-Krzesinski, Noshir Contractor, Stephen M Fiore, Kara L Hall, Cathleen Kane, Joann Keyton, Julie Thompson Klein, Bonnie Spring, Daniel Stokols, and William Trochim. 2011. Mapping a research agenda for the science of team science. Research Evaluation 20, 2 (2011), 145--158.

[20]

Ohad Fried, Stephen DiVerdi, Maciej Halber, Elena Sizikova, and Adam Finkelstein. 2015. IsoMatch: Creating informative grid layouts. In Computer graphics forum, Vol. 34. Wiley Online Library, 155--166.

Digital Library

[21]

Emden R Gansner, Yifan Hu, and Stephen Kobourov. 2010. GMap: Visualizing graphs and clusters as maps. In Visualization Symposium (PacificVis), 2010 IEEE Pacific. IEEE, 201--208.

[22]

Elena Garces, Aseem Agarwala, Aaron Hertzmann, and Diego Gutierrez. 2017. Style-based exploration of illustration datasets. Multimedia Tools and Applications 76, 11 (2017), 13067--13086.

Digital Library

[23]

Alyssa Glass, Deborah L McGuinness, and Michael Wolverton. 2008. Toward establishing trust in adaptive agents. In Proceedings of the 13th international conference on Intelligent user interfaces. ACM, 227--236.

Digital Library

[24]

Nitesh Goyal, Gilly Leshed, Dan Cosley, and Susan R Fussell. 2014. Effects of implicit sharing in collaborative analysis. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 129--138.

Digital Library

[25]

Raja Gumienny, Steven P Dow, and Christoph Meinel. 2014. Supporting the synthesis of information in design teams. In Proceedings of the 2014 conference on Designing interactive systems. ACM, 463--472.

Digital Library

[26]

Steven Hansen, N Hari Narayanan, and Mary Hegarty. 2002. Designing educationally effective algorithm visualizations. Journal of Visual Languages & Computing 13, 3 (2002), 291--317.

[27]

Gunnar Harboe, Jonas Minke, Ioana Ilea, and Elaine M Huang. 2012. Computer support for collaborative data analysis: augmenting paper affinity diagrams. In Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work. ACM, 1179--1182.

Digital Library

[28]

Jason L Harman, John O'Donovan, Tarek Abdelzaher, and Cleotilde Gonzalez. 2014. Dynamics of human trust in recommender systems. In Proceedings of the 8th ACM Conference on Recommender systems. ACM, 305--308.

Digital Library

[29]

Nicholas Jenkins, Michael Bloor, Jan Fischer, Lee Berney, and Joanne Neale. 2010. Putting it in context: the use of vignettes in qualitative interviewing. Qualitative Research 10, 2 (2010), 175--198.

[30]

Benjamin King. 1967. Step-wise clustering procedures. J. Amer. Statist. Assoc. 62, 317 (1967), 86--101.

[31]

René F Kizilcec. 2016. How much information?: Effects of transparency on trust in an algorithmic interface. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 2390--2395.

Digital Library

[32]

Bart P Knijnenburg, Svetlin Bostandjiev, John O'Donovan, and Alfred Kobsa. 2012. Inspectability and control in social recommenders. In Proceedings of the sixth ACM conference on Recommender systems. ACM, 43--50.

Digital Library

[33]

Brian Y Lim, Anind K Dey, and Daniel Avrahami. 2009. Why and why not explanations improve the intelligibility of context-aware intelligent systems. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 2119--2128.

Digital Library

[34]

Zachary C Lipton. 2016. The mythos of model interpretability. arXiv preprint arXiv:1606.03490 (2016).

[35]

Paulo JG Lisboa. 2013. Interpretability in Machine Learning--Principles and Practice. In International Workshop on Fuzzy Logic and Applications. Springer, Cham, 15--21.

[36]

Duen-Ren Liu, Chin-Hui Lai, and Hsuan Chiu. 2011. Sequence-based trust in collaborative filtering for document recommendation. International Journal of Human-Computer Studies 69, 9 (2011), 587--601.

Digital Library

[37]

Wendy L Martinez, Angel R Martinez, Angel Martinez, and Jeffrey Solka. 2010. Exploratory data analysis with MATLAB. CRC Press.

Digital Library

[38]

James Munkres. 1957. Algorithms for the assignment and transportation problems. Journal of the society for industrial and applied mathematics 5, 1 (1957), 32--38.

[39]

Joseph Novak. 1991. Clarify with concept maps. The science teacher 58, 7 (1991), 44.

[40]

Joseph D Novak and Alberto J Cañas. 2008. The theory underlying concept maps and how to construct and use them. (2008).

[41]

Joseph D Novak and Dismas Musonda. 1991. A twelve-year longitudinal study of science concept learning. American Educational Research Journal 28, 1 (1991), 117--153.

[42]

Conor Nugent and Pádraig Cunningham. 2005. A case-based explanation system for black-box systems. Artificial Intelligence Review 24, 2 (2005), 163--178.

Digital Library

[43]

Stefano Padilla, Thomas S Methven, David W Corne, and Mike J Chantler. 2014. Hot topics in CHI: trend maps for visualising research. In CHI'14 Extended Abstracts on Human Factors in Computing Systems. ACM, 815--824.

Digital Library

[44]

Stefano Padilla, Thomas S Methven, David A Robb, and Mike J Chantler. 2017. Understanding Concept Maps: A Closer Look at How People Organise Ideas. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. ACM, 815--827.

Digital Library

[45]

Michael Quinn Patton. 1990. Qualitative evaluation and research methods. SAGE Publications, inc.

[46]

Wolter Pieters. 2006. Acceptance of voting technology: between confidence and trust. In iTrust. Springer, 283--297.

Digital Library

[47]

Wolter Pieters. 2011. Explanation and trust: what to tell the user in security and AI? Ethics and information technology 13, 1 (2011), 53--64.

Digital Library

[48]

Roberto Pinho, Maria Cristina F de Oliveira, and Alneu de A Lopes. 2009. Incremental board: a grid-based space for visualizing dynamic data sets. In Proceedings of the 2009 ACM symposium on Applied Computing. ACM, 1757--1764.

Digital Library

[49]

Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. Why should i trust you?: Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 1135--1144.

Digital Library

[50]

David Silverman. 2013. Doing qualitative research: A practical handbook. SAGE Publications Limited.

[51]

Robert R Sokal. 1958. A statistical method for evaluating systematic relationship. University of Kansas science bulletin 28 (1958), 1409--1438.

[52]

Justin Solomon, Gabriel Peyré, Vladimir G Kim, and Suvrit Sra. 2016. Entropic metric alignment for correspondence problems. ACM Transactions on Graphics (TOG) 35, 4 (2016), 72.

Digital Library

[53]

Anselm L Strauss. 1987. Qualitative analysis for social scientists. Cambridge University Press.

[54]

Pang-Ning Tan, Michael Steinbach, and Vipin Kumar. 2005. Introduction to data mining. 1st. (2005).

[55]

Joshua B Tenenbaum, Vin De Silva, and John C Langford. 2000. A global geometric framework for nonlinear dimensionality reduction. science 290, 5500 (2000), 2319--2323.

[56]

William MK Trochim. 1989. An introduction to concept mapping for planning and evaluation. Evaluation and program planning 12, 1 (1989), 1--16.

[57]

William M Trochim. 2017. Hindsight is 20/20: Reflections on the evolution of concept mapping. Evaluation and program planning 60 (2017), 176--185.

[58]

Tamara Van Gog, Liesbeth Kester, Fleurie Nievelstein, Bas Giesbers, and Fred Paas. 2009. Uncovering cognitive processes: Different techniques that can contribute to cognitive load research and instruction. Computers in Human Behavior 25, 2 (2009), 325--331.

Digital Library

[59]

Colin Ware. 2012. Information visualization: perception for design. Elsevier.

Digital Library

[60]

Kelly D Wason, Michael J Polonsky, and Michael R Hyman. 2002. Designing vignette studies in marketing. Australasian Marketing Journal (AMJ) 10, 3 (2002), 41--58.

[61]

Tom Wengraf. 2001. Qualitative research interviewing: Biographic narrative and semi-structured methods. Sage.

[62]

James A Wise, James J Thomas, Kelly Pennock, David Lantrip, Marc Pottier, Anne Schur, and Vern Crow. 1995. Visualizing the non-visual: Spatial analysis and interaction with information from text documents. In Information Visualization, 1995. Proceedings. IEEE, 51--58.

Digital Library

[63]

Iwan GJH Wopereis, Paul A Kirschner, Fred Paas, Slavi Stoyanov, and Maaike Hendriks. 2005. Failure and success factors of educational ICT projects: a group concept mapping approach. British Journal of Educational Technology 36, 4 (2005), 681--684.

Cited By

Rong YLeemann TNguyen TFiedler LQian PUnhelkar VSeidel TKasneci GKasneci E(2024)Towards Human-Centered Explainable AI: A Survey of User Studies for Model ExplanationsIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2023.333184646:4(2104-2122)Online publication date: Apr-2024
https://doi.org/10.1109/TPAMI.2023.3331846
Carrera DOvienmhada UHussein SSoden R(2023)The Unseen Landscape of Abolitionism: Examining the Role of Digital Maps in Grassroots OrganizingProceedings of the ACM on Human-Computer Interaction10.1145/36102147:CSCW2(1-29)Online publication date: 4-Oct-2023
https://dl.acm.org/doi/10.1145/3610214
van Berkel NSarsenbayeva ZGoncalves J(2023)The methodology of studying fairness perceptions in Artificial IntelligenceInternational Journal of Human-Computer Studies10.1016/j.ijhcs.2022.102954170:COnline publication date: 8-Feb-2023
https://dl.acm.org/doi/10.1016/j.ijhcs.2022.102954
Show More Cited By

Index Terms

Improving User Confidence in Concept Maps: Exploring Data Driven Explanations
1. Human-centered computing
  1. Human computer interaction (HCI)

Recommendations

Perspectives on user experience for a nation-wide senior telehealth program
HCI '16: Proceedings of the 30th International BCS Human Computer Interaction Conference: Fusion!

Prior evidence shows that senior people accept technology and enjoy self-monitoring. Home telehealth support increases their confidence and sense of security. The use of communication technologies plays an important role avoiding isolation and ...
Using a concept map to evaluate exploratory search
IIiX '10: Proceedings of the third symposium on Information interaction in context

Users often conduct exploratory searches on the Web to solve complex problems, but how for evaluating the effectiveness of a search, especially in a user-centered approach, remains an open question. We propose a user-centered method to evaluate the ...
Investigating User Confidence for Uncertainty Presentation in Predictive Decision Making
OzCHI '15: Proceedings of the Annual Meeting of the Australian Special Interest Group for Computer Human Interaction

Machine Learning (ML) based decision support systems are often like a black box to non-expert users. Here user's confidence becomes critical for effective decision making and maintaining trust in the system. We find that user confidence varies ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '18: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems

April 2018

8489 pages

ISBN:9781450356206

DOI:10.1145/3173574

General Chairs:
Regan Mandryk
University of Saskatchewan, Canada
,
Mark Hancock
University of Waterloo, Canada
,
Program Chairs:
Mark Perry
Brunel University London, UK
,
Anna Cox
University College London, UK

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 21 April 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

EPSRC
Heriot-Watt University

Conference

CHI '18

Sponsor:

SIGCHI

CHI '18: CHI Conference on Human Factors in Computing Systems

April 21 - 26, 2018

Montreal QC, Canada

Acceptance Rates

CHI '18 Paper Acceptance Rate 666 of 2,590 submissions, 26%;

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

14
Total Citations
View Citations
546
Total Downloads

Downloads (Last 12 months)47
Downloads (Last 6 weeks)11

Reflects downloads up to 18 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Rong YLeemann TNguyen TFiedler LQian PUnhelkar VSeidel TKasneci GKasneci E(2024)Towards Human-Centered Explainable AI: A Survey of User Studies for Model ExplanationsIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2023.333184646:4(2104-2122)Online publication date: Apr-2024
https://doi.org/10.1109/TPAMI.2023.3331846
Carrera DOvienmhada UHussein SSoden R(2023)The Unseen Landscape of Abolitionism: Examining the Role of Digital Maps in Grassroots OrganizingProceedings of the ACM on Human-Computer Interaction10.1145/36102147:CSCW2(1-29)Online publication date: 4-Oct-2023
https://dl.acm.org/doi/10.1145/3610214
van Berkel NSarsenbayeva ZGoncalves J(2023)The methodology of studying fairness perceptions in Artificial IntelligenceInternational Journal of Human-Computer Studies10.1016/j.ijhcs.2022.102954170:COnline publication date: 8-Feb-2023
https://dl.acm.org/doi/10.1016/j.ijhcs.2022.102954
Liu CHuang ZLiu DZhou SPeng ZMa X(2022)PlanHelperProceedings of the ACM on Human-Computer Interaction10.1145/35555556:CSCW2(1-26)Online publication date: 11-Nov-2022
https://dl.acm.org/doi/10.1145/3555555
Parra-Ullauri JGarcía-Domínguez ABencomo NGarcia-Paucar LKühn TSousa V(2022)History-aware explanationsProceedings of the 25th International Conference on Model Driven Engineering Languages and Systems: Companion Proceedings10.1145/3550356.3561538(286-295)Online publication date: 23-Oct-2022
https://dl.acm.org/doi/10.1145/3550356.3561538
Brownlee AEpitropakis MMulder JPaelinck MBurke E(2022)A systematic approach to parameter optimization and its application to flight schedule simulation softwareJournal of Heuristics10.1007/s10732-022-09501-828:4(509-538)Online publication date: 5-Jul-2022
https://doi.org/10.1007/s10732-022-09501-8
Chiyah Garcia FSmith SLopes JRamamoorthy SHastie HBethel CPaiva ABroadbent EFeil-Seifer DSzafir D(2021)Self-Explainable Robots in Remote EnvironmentsCompanion of the 2021 ACM/IEEE International Conference on Human-Robot Interaction10.1145/3434074.3447275(662-664)Online publication date: 8-Mar-2021
https://dl.acm.org/doi/10.1145/3434074.3447275
Parra-Ullauri JGarcía-Domínguez ABencomo NZheng CZhen CBoubeta-Puig JOrtiz GYang S(2021)Event-driven temporal models for explanations - ETeMoX: explaining reinforcement learningSoftware and Systems Modeling10.1007/s10270-021-00952-421:3(1091-1113)Online publication date: 18-Dec-2021
https://doi.org/10.1007/s10270-021-00952-4
Wallace ABrownlee ACairns D(2021)Towards Explaining Metaheuristic Solution Quality by Data Mining Surrogate Fitness Models for Importance of VariablesArtificial Intelligence XXXVIII10.1007/978-3-030-91100-3_5(58-72)Online publication date: 6-Dec-2021
https://doi.org/10.1007/978-3-030-91100-3_5
Naiseh MJiang NMa JAli R(2020)Explainable Recommendations in Intelligent Systems: Delivery Methods, Modalities and RisksResearch Challenges in Information Science10.1007/978-3-030-50316-1_13(212-228)Online publication date: 25-Jun-2020
https://doi.org/10.1007/978-3-030-50316-1_13
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents