Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
10.1145/3411764.3445444acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
research-article

Causal Perception in Question-Answering Systems

Published: 07 May 2021 Publication History

Abstract

Root cause analysis is a common data analysis task. While question-answering systems enable people to easily articulate a why question (e.g., why students in Massachusetts have high ACT Math scores on average) and obtain an answer, these systems often produce questionable causal claims. To investigate how such claims might mislead users, we conducted two crowdsourced experiments to study the impact of showing different information on user perceptions of a question-answering system. We found that in a system that occasionally provided unreasonable responses, showing a scatterplot increased the plausibility of unreasonable causal claims. Also, simply warning participants that correlation is not causation seemed to lead participants to accept reasonable causal claims more cautiously. We observed a strong tendency among participants to associate correlation with causation. Yet, the warning appeared to reduce the tendency. Grounded in the findings, we propose ways to reduce the illusion of causality when using question-answering systems.

Supplementary Material

VTT File (3411764.3445444_videopreviewcaptions.vtt)
Supplementary Materials (3411764.3445444_supplementalmaterials.zip)
MP4 File (3411764.3445444_videopreview.mp4)
Preview video

References

[1]
Outlier AI. 2017. How to Conduct a Proper Root Cause Analysis. https://towardsdatascience.com/how-to-conduct-a-proper-root-cause-analysis-789b9847f84b
[2]
Robert Amar, James Eagan, and John Stasko. 2005. Low-Level Components of Analytic Activity in Information Visualization. In IEEE Symposium on Information Visualization. IEEE, 111–117. https://doi.org/10.1109/INFVIS.2005.1532136
[3]
Apple. 2020. Siri - Apple. https://www.apple.com/siri/
[4]
Zahra Ashktorab, Mohit Jain, Q Vera Liao, and Justin D Weisz. 2019. Resilient Chatbots: Repair Strategy Preferences for Conversational Breakdowns. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, 1–12. https://doi.org/10.1145/3290605.3300484
[5]
William M Bolstad and James M Curran. 2016. Introduction to Bayesian Statistics. John Wiley & Sons.
[6]
Michelle A Borkin, Zoya Bylinskii, Nam Wook Kim, Constance May Bainbridge, Chelsea S Yeh, Daniel Borkin, Hanspeter Pfister, and Aude Oliva. 2015. Beyond Memorability: Visualization Recognition and Recall. IEEE Transactions on Visualization and Computer Graphics 22, 1(2015), 519–528. https://doi.org/10.1109/TVCG.2015.2467732
[7]
Jeremy Boy, Anshul Vikram Pandey, John Emerson, Margaret Satterthwaite, Oded Nov, and Enrico Bertini. 2017. Showing People Behind Data: Does Anthropomorphizing Visualizations Elicit More Empathy for Human Rights Data?. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. ACM, 5462–5474. https://doi.org/10.1145/3025453.3025512
[8]
United States Census Bureau. 2020. Census Bureau. https://www.census.gov
[9]
Hao-Fei Cheng, Ruotong Wang, Zheng Zhang, Fiona O’Connell, Terrance Gray, F Maxwell Harper, and Haiyi Zhu. 2019. Explaining Decision-Making Algorithms Through UI: Strategies to Help Non-Expert Stakeholders. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, 1–12. https://doi.org/10.1145/3290605.3300789
[10]
William S Cleveland, Persi Diaconis, and Robert McGill. 1982. Variables on Scatterplots Look More Highly Correlated When the Scales Are Increased. Science 216, 4550 (1982), 1138–1141. https://doi.org/10.1126/science.216.4550.1138
[11]
Wharton County Junior College. 2020. Root Cause Analysis Training. https://www.wcjc.edu/Programs/continuing-education/root-cause.aspx
[12]
Michael Correll. 2019. Ethical Dimensions of Visualization Research. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, 1–13. https://doi.org/10.1145/3290605.3300418
[13]
Michael Correll, Enrico Bertini, and Steven Franconeri. 2020. Truncating the Y-Axis: Threat or Menace?. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. ACM, 1–12. https://doi.org/10.1145/3313831.3376222
[14]
Michael Correll and Jeffrey Heer. 2017. Black Hat Visualization. In Workshop on Dealing with Cognitive Biases in Visualisations (DECISIVe). https://decisive-workshop.dbvis.de/wp-content/uploads/2017/09/0115-paper.pdf
[15]
Coursera. 2020. Root Cause Analysis - Root Cause Analysis | Coursera. https://www.coursera.org/lecture/six-sigma-analyze/root-cause-analysis-w01Qj
[16]
Kedar Dhamdhere, Kevin S McCurley, Ralfi Nahmias, Mukund Sundararajan, and Qiqi Yan. 2017. Analyza: Exploring Data With Conversation. In Proceedings of the 22nd International Conference on Intelligent User Interfaces. ACM, 493–504. https://doi.org/10.1145/3025171.3025227
[17]
Evanthia Dimara, Gilles Bailly, Anastasia Bezerianos, and Steven Franconeri. 2018. Mitigating the Attraction Effect With Visualizations. IEEE Transactions on Visualization and Computer Graphics 25, 1(2018), 850–860. https://doi.org/10.1109/TVCG.2018.2865233
[18]
Evanthia Dimara, Steven Franconeri, Catherine Plaisant, Anastasia Bezerianos, and Pierre Dragicevic. 2020. A task-based taxonomy of cognitive biases for information visualization. IEEE Transactions on Visualization and Computer Graphics 26, 2(2020), 1413–1432. https://doi.org/10.1109/TVCG.2018.2872577
[19]
Serge Egelman, Lorrie Faith Cranor, and Jason Hong. 2008. You’ve Been Warned: An Empirical Study of the Effectiveness of Web Browser Phishing Warnings. In Proceedings of the 2008 CHI Conference on Human Factors in Computing Systems. ACM, 1065–1074. https://doi.org/10.1145/1357054.1357219
[20]
Ethan Fast, Binbin Chen, Julia Mendelsohn, Jonathan Bassen, and Michael S Bernstein. 2018. Iris: A Conversational Agent for Complex Tasks. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 1–13. https://doi.org/10.1145/3173574.3174047
[21]
Asbjørn Følstad and Petter Bae Brandtzæg. 2017. Chatbots and the New World of HCI. Interactions 24, 4 (2017), 38–42. https://doi.org/10.1145/3085558
[22]
National Center for Education Statistics. 2020. National Center for Education Statistics (NCES) Home Page, a part of the U.S. Department of Education. https://nces.ed.gov
[23]
Kaiser Family Foundation. 2020. KFF - Health Policy Analysis, Polling and Journalism. https://www.kff.org
[24]
Tong Gao, Mira Dontcheva, Eytan Adar, Zhicheng Liu, and Karrie G Karahalios. 2015. Datatone: Managing Ambiguity in Natural Language Interfaces for Data Visualization. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology. ACM, 489–500. https://doi.org/10.1145/2807442.2807478
[25]
Marti Hearst and Melanie Tory. 2019. Would You Like a Chart With That? Incorporating Visualizations Into Conversational Interfaces. In 2019 IEEE Visualization Conference (VIS). IEEE, 1–5. https://doi.org/10.1109/VISUAL.2019.89337668
[26]
Marti Hearst, Melanie Tory, and Vidya Setlur. 2019. Toward Interface Defaults for Vague Modifiers in Natural Language Interfaces for Visual Analysis. In 2019 IEEE Visualization Conference (VIS). IEEE, 21–25. https://doi.org/10.1109/VISUAL.2019.8933569
[27]
Jeffrey Heer and Maneesh Agrawala. 2006. Multi-Scale Banking to 45 Degrees. IEEE Transactions on Visualization and Computer Graphics 12, 5(2006), 701–708. https://doi.org/10.1109/TVCG.2006.163
[28]
Elizabeth Hellier, Daniel B Wright, Judy Edworthy, and Stephen Newstead. 2000. On the Stability of the Arousal Strength of Warning Signal Words. Applied Cognitive Psychology: The Official Journal of the Society for Applied Research in Memory and Cognition 14, 6 (2000), 577–592. https://doi.org/10.1002/1099-0720(200011/12)14:6<577::AID-ACP682>3.0.CO;2-A
[29]
Jonathan L Herlocker, Joseph A Konstan, and John Riedl. 2000. Explaining Collaborative Filtering Recommendations. In Proceedings of the 2000 ACM Conference on Computer Supported Cooperative Work. ACM, 241–250. https://doi.org/10.1145/358916.358995
[30]
Enamul Hoque, Vidya Setlur, Melanie Tory, and Isaac Dykeman. 2017. Applying Pragmatics Principles for Interaction With Visual Analytics. IEEE Transactions on Visualization and Computer Graphics 24, 1(2017), 309–318. https://doi.org/10.1109/TVCG.2017.2744684
[31]
Jessica Hullman and Nick Diakopoulos. 2011. Visualization Rhetoric: Framing Effects in Narrative Visualization. IEEE Transactions on Visualization and Computer Graphics 17, 12(2011), 2231–2240. https://doi.org/10.1109/TVCG.2011.255
[32]
Yea-Seul Kim, Katharina Reinecke, and Jessica Hullman. 2017. Data Through Others’ Eyes: The Impact of Visualizing Others’ Expectations on Visualization Interpretation. IEEE Transactions on Visualization and Computer Graphics 24, 1(2017), 760–769. https://doi.org/10.1109/TVCG.2017.2745240
[33]
Gary Klein, Brian Moon, and Robert R Hoffman. 2006. Making Sense of Sensemaking 1: Alternative Perspectives. IEEE Intelligent Systems 21, 4 (2006), 70–73. https://doi.org/10.1109/MIS.2006.75
[34]
Gary Klein, Brian Moon, and Robert R Hoffman. 2006. Making Sense of Sensemaking 2: A Macrocognitive Model. IEEE Intelligent Systems 21, 5 (2006), 88–92. https://doi.org/10.1109/MIS.2006.100
[35]
Ha-Kyung Kong, Zhicheng Liu, and Karrie Karahalios. 2018. Frames and Slants in Titles of Visualizations on Controversial Topics. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 1–12. https://doi.org/10.1145/3173574.3174012
[36]
Ha-Kyung Kong, Zhicheng Liu, and Karrie Karahalios. 2019. Trust and Recall of Information Across Varying Degrees of Title-Visualization Misalignment. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, 1–13. https://doi.org/10.1145/3290605.3300576
[37]
Po-Ming Law, Rahul C Basole, and Yanhong Wu. 2018. Duet: Helping Data Analysis Novices Conduct Pairwise Comparisons by Minimal Specification. IEEE Transactions on Visualization and Computer Graphics 25, 1(2018), 427–437. https://doi.org/10.1109/TVCG.2018.2864526
[38]
Po-Ming Law, Subhajit Das, and Rahul C Basole. 2019. Comparing Apples and Oranges: Taxonomy and Design of Pairwise Comparisons within Tabular Data. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, 1–12. https://doi.org/10.1145/3290605.3300409
[39]
Po-Ming Law, Alex Endert, and John Stasko. 2020. Characterizing Automated Data Insights. arXiv preprint arXiv:2008.13060(2020). https://arxiv.org/abs/2008.13060
[40]
Victoria Lee. 2020. Why We Quarantine: A Data Driven Love Letter to You and the Loves of Your Life. https://medium.com/swlh/why-we-quarantine-a-data-driven-love-letter-to-you-and-the-loves-of-your-life-c19de2bca87f
[41]
Erich L Lehmann. 2006. Nonparametrics: Statistical Methods Based on Ranks. Springer-Verlag, New York, NY, USA.
[42]
Stephan Lewandowsky, Ullrich KH Ecker, Colleen M Seifert, Norbert Schwarz, and John Cook. 2012. Misinformation and Its Correction: Continued Influence and Successful Debiasing. Psychological Science in the Public Interest 13, 3 (2012), 106–131. https://doi.org/10.1177/1529100612451018
[43]
Q Vera Liao, Matthew Davis, Werner Geyer, Michael Muller, and N Sadat Shami. 2016. What Can You Do? Studying Social-Agent Orientation and Agent Proactive Interactions With an Agent for Employees. In Proceedings of the 2016 ACM Conference on Designing Interactive Systems. ACM, 264–275. https://doi.org/10.1145/2901790.2901842
[44]
Q Vera Liao, Muhammed Mas-ud Hussain, Praveen Chandar, Matthew Davis, Yasaman Khazaeni, Marco Patricio Crasso, Dakuo Wang, Michael Muller, N Sadat Shami, and Werner Geyer. 2018. All Work and No Play? Conversations With a Question-And-Answer Chatbot in the Wild. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 1–13. https://doi.org/10.1145/3173574.3173577
[45]
Zhicheng Liu and Jeffrey Heer. 2014. The Effects of Interactive Latency on Exploratory Visual Analysis. IEEE Transactions on Visualization and Computer Graphics 20, 12(2014), 2122–2131. https://doi.org/10.1109/TVCG.2014.2346452
[46]
m00nlight Wang. 2018. Income Inequality Analysis and Visualization. https://medium.com/@m00nlight/income-inequality-analysis-and-visualization-f688a4fc6609
[47]
Helena Matute, Fernando Blanco, Ion Yarritu, Marcos Díaz-Lago, Miguel A Vadillo, and Itxaso Barberia. 2015. Illusions of Causality: How They Bias Our Everyday Thinking and How They Could Be Reduced. Frontiers in Psychology 6 (2015), 888. https://doi.org/10.3389/fpsyg.2015.00888
[48]
Luana Micallef, Gregorio Palmas, Antti Oulasvirta, and Tino Weinkauf. 2017. Towards Perceptual Optimization of the Visual Design of Scatterplots. IEEE Transactions on Visualization and Computer Graphics 23, 6(2017), 1588–1599. https://doi.org/10.1109/TVCG.2017.2674978
[49]
Andrew Vande Moere, Martin Tomitsch, Christoph Wimmer, Boesch Christoph, and Thomas Grechenig. 2012. Evaluating the Effect of Style in Information Visualization. IEEE Transactions on Visualization and Computer Graphics 18, 12(2012), 2739–2748. https://doi.org/10.1109/TVCG.2012.221
[50]
Makeover Monday. 2020. Makeover Monday | a Weekly Social Data Project. https://www.makeovermonday.co.uk
[51]
Anshul Vikram Pandey, Anjali Manivannan, Oded Nov, Margaret Satterthwaite, and Enrico Bertini. 2014. The Persuasive Power of Data Visualization. IEEE Transactions on Visualization and Computer Graphics 20, 12(2014), 2211–2220. https://doi.org/10.1109/TVCG.2014.2346419
[52]
Anshul Vikram Pandey, Katharina Rall, Margaret L Satterthwaite, Oded Nov, and Enrico Bertini. 2015. How Deceptive Are Deceptive Visualizations? an Empirical Analysis of Common Distortion Techniques. In Proceedings of the 2015 CHI Conference on Human Factors in Computing Systems. ACM, 1469–1478. https://doi.org/10.1145/2702123.2702608
[53]
Pearl Pu, Li Chen, and Rong Hu. 2011. A User-Centric Evaluation Framework for Recommender System. In Proceedings of the Fifth ACM Conference on Recommender Systems. ACM, 157–164. https://doi.org/10.1145/2043932.2043962
[54]
Jacob Ritchie, Daniel Wigdor, and Fann Chevalier. 2019. A Lie Reveals the Truth: Quasimodes for Task-Aligned Data Presentation. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, 1–13. https://doi.org/10.1145/3290605.3300423
[55]
Julia M Rohrer. 2018. Thinking Clearly About Correlations and Causation: Graphical Causal Models for Observational Data. Advances in Methods and Practices in Psychological Science 1, 1(2018), 27–42. https://doi.org/10.1177/2515245917745629
[56]
Dominik Sacha, Andreas Stoffel, Florian Stoffel, Bum Chul Kwon, Geoffrey Ellis, and Daniel A Keim. 2014. Knowledge Generation Model for Visual Analytics. IEEE Transactions on Visualization and Computer Graphics 20, 12(2014), 1604–1613. https://doi.org/10.1109/TVCG.2014.2346481
[57]
Alper Sarikaya and Michael Gleicher. 2017. Scatterplots: Tasks, Data, and Designs. IEEE Transactions on Visualization and Computer Graphics 24, 1(2017), 402–412. https://doi.org/10.1109/TVCG.2017.2744184
[58]
Vidya Setlur, Sarah E Battersby, Melanie Tory, Rich Gossweiler, and Angel X Chang. 2016. Eviza: A Natural Language Interface for Visual Analysis. In Proceedings of the 29th Annual ACM Symposium on User Interface Software & Technology. ACM, 365–377. https://doi.org/10.1145/2984511.2984588
[59]
Rashmi Sinha and Kirsten Swearingen. 2002. The Role of Transparency in Recommender Systems. In Extended Abstracts of the 2002 CHI Conference on Human Factors in Computing Systems. ACM, 830–831. https://doi.org/10.1145/506443.506619
[60]
Arjun Srinivasan, Steven M Drucker, Alex Endert, and John Stasko. 2018. Augmenting Visualizations With Interactive Data Facts to Facilitate Interpretation and Communication. IEEE Transactions on Visualization and Computer Graphics 25, 1(2018), 672–681. https://doi.org/10.1109/TVCG.2018.2865145
[61]
Arjun Srinivasan and John Stasko. 2017. Orko: Facilitating Multimodal Interaction for Visual Exploration and Analysis of Networks. IEEE Transactions on Visualization and Computer Graphics 24, 1(2017), 511–521. https://doi.org/10.1109/TVCG.2017.2745219
[62]
Tableau. 2019. Explain Data Internals: Automated Bayesian Modeling | Tableau Conference 2019. https://tc19.tableau.com/learn/sessions/explain-data-internals-automated-bayesian-modeling?_ga=2.242994050.1845292459.1583776901-580893601.1583776901&_fsi=H59ZIxRV
[63]
Tableau. 2019. Inspect a View using Explain Data – Tableau. https://help.tableau.com/current/pro/desktop/en-us/explain_data.htm
[64]
Tableau. 2020. Explain Data | Tableau Software. https://www.tableau.com/products/new-features/explain-data
[65]
Andre Calero Valdez, Martina Ziefle, and Michael Sedlmair. 2017. Priming and Anchoring Effects in Visualization. IEEE Transactions on Visualization and Computer Graphics 24, 1(2017), 584–594. https://doi.org/10.1109/TVCG.2017.2744138
[66]
Jun Wang and Klaus Mueller. 2015. The Visual Causality Analyst: An Interactive Interface for Causal Reasoning. IEEE Transactions on Visualization and Computer Graphics 22, 1(2015), 230–239. https://doi.org/10.1109/TVCG.2015.2467931
[67]
Michael S Wogalter. 2006. Communication-Human Information Processing (C-HIP) Model. Handbook of warnings(2006), 51–61.
[68]
Cindy Xiong, Joel Shapiro, Jessica Hullman, and Steven Franconeri. 2019. Illusion of Causality in Visualized Data. IEEE Transactions on Visualization and Computer Graphics 26, 1(2019), 853–862. https://doi.org/10.1109/TVCG.2019.2934399
[69]
Cindy Xiong, Lisanne van Weelden, and Steven Franconeri. 2019. The Curse of Knowledge in Visual Data Communication. IEEE Transactions on Visualization and Computer Graphics (2019). https://doi.org/10.1109/TVCG.2019.2917689
[70]
Bowen Yu and Cláudio T Silva. 2019. Flowsense: A Natural Language Interface for Visual Data Exploration Within a Dataflow System. IEEE Transactions on Visualization and Computer Graphics 26, 1(2019), 1–11. https://doi.org/10.1109/TVCG.2019.2934668
[71]
Mingqian Zhao, Huamin Qu, and Michael Sedlmair. 2019. Neighborhood Perception in Bar Charts. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, 1–12. https://doi.org/10.1145/3290605.3300462
[72]
Qiyu Zhi and Ronald Metoyer. 2020. GameBot: A Visualization-Augmented Chatbot for Sports Game. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems. ACM, 1–7. https://doi.org/10.1145/3334480.3382794

Cited By

View all
  • (2023)On Chatbots for Visual Exploratory Data Analysis2023 IEEE International Conference on Big Data (BigData)10.1109/BigData59044.2023.10386335(5924-5929)Online publication date: 15-Dec-2023
  • (2022)Bias-Aware Design for Informed Decisions: Raising Awareness of Self-Selection Bias in User Ratings and ReviewsProceedings of the ACM on Human-Computer Interaction10.1145/35555976:CSCW2(1-31)Online publication date: 11-Nov-2022
  • (2022)Evaluating the Effect of Enhanced Text-Visualization Integration on Combating Misinformation in Data Story2022 IEEE 15th Pacific Visualization Symposium (PacificVis)10.1109/PacificVis53943.2022.00023(141-150)Online publication date: Apr-2022

Index Terms

  1. Causal Perception in Question-Answering Systems
    Index terms have been assigned to the content through auto-classification.

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    CHI '21: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
    May 2021
    10862 pages
    ISBN:9781450380966
    DOI:10.1145/3411764
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 07 May 2021

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. correlation and causation
    2. question answering

    Qualifiers

    • Research-article
    • Research
    • Refereed limited

    Conference

    CHI '21
    Sponsor:

    Acceptance Rates

    Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)59
    • Downloads (Last 6 weeks)3
    Reflects downloads up to 15 Oct 2024

    Other Metrics

    Citations

    Cited By

    View all
    • (2023)On Chatbots for Visual Exploratory Data Analysis2023 IEEE International Conference on Big Data (BigData)10.1109/BigData59044.2023.10386335(5924-5929)Online publication date: 15-Dec-2023
    • (2022)Bias-Aware Design for Informed Decisions: Raising Awareness of Self-Selection Bias in User Ratings and ReviewsProceedings of the ACM on Human-Computer Interaction10.1145/35555976:CSCW2(1-31)Online publication date: 11-Nov-2022
    • (2022)Evaluating the Effect of Enhanced Text-Visualization Integration on Combating Misinformation in Data Story2022 IEEE 15th Pacific Visualization Symposium (PacificVis)10.1109/PacificVis53943.2022.00023(141-150)Online publication date: Apr-2022

    View Options

    Get Access

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    HTML Format

    View this article in HTML Format.

    HTML Format

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media