research-article

When Algorithms Err: Differential Impact of Early vs. Late Errors on Users’ Reliance on Algorithms

Authors:

Jingjing ZhangAuthors Info & Claims

ACM Transactions on Computer-Human Interaction, Volume 30, Issue 1

Article No.: 14, Pages 1 - 36

https://doi.org/10.1145/3557889

Published: 18 March 2023 Publication History

Abstract

Errors are a natural part of predictive algorithms, but may discourage users from relying on algorithms. We conduct two experiments to demonstrate that reliance on a predictive algorithm following a substantial error is affected by (i) when the error occurs and (ii) how the algorithm is used in the decision-making process. We find that the impact of an error on reliance depends on whether the error occurs early (i.e., when users first start using the algorithm) or late (i.e., after users have used the algorithm for an extended period). While an early error results in substantial and persistent reliance reduction, a late error affects reliance only temporarily and to a lesser extent. However, when users have more control over how to use the algorithm’s predictions, error timing ceases to have a significant impact. Our work advances the understanding of algorithm aversion and informs the practical design of algorithmic decision-making systems.

References

[1]

Ahmed Abbasi, Hsinchun Chen, and Arab Salem. 2008. Sentiment analysis in multiple languages: Feature selection for opinion classification in web forums. ACM Transactions on Information Systems 26, 3 (2008), 1–34.

Digital Library

[2]

Michael D. Abràmoff, Philip T. Lavin, Michele Birch, Nilay Shah, and James C. Folk. 2018. Pivotal trial of an autonomous AI-based diagnostic system for detection of diabetic retinopathy in primary care offices. NPJ Digital Medicine 1, 1 (2018), 1–8.

[3]

Ritu Agarwal and Jayesh Prasad. 1998. A conceptual and operational definition of personal innovativeness in the domain of information technology. Information Systems Research 9, 2 (1998), 204–215.

Digital Library

[4]

Ashish Arora, Jonathan P. Caulkins, and Rahul Telang. 2006. Research note: Sell first, fix later: Impact of patching on software quality. Management Science 52, 3 (2006), 465–471.

Digital Library

[5]

Solomon E. Asch. 1946. Forming impressions of personality.The Journal of Abnormal and Social Psychology 41, 3 (1946), 258.

[6]

Timothy W. Bickmore and Rosalind W. Picard. 2005. Establishing and maintaining long-term human-computer relationships. ACM Transactions on Computer-Human Interaction 12, 2 (2005), 293–327.

Digital Library

[7]

John Bohannon. 2016. Mechanical Turk upends social sciences. Science 352, 6291 (2016), 1263–1264.

[8]

Noah Castelo, Maarten W. Bos, and Donald R. Lehmann. 2019. Task-dependent algorithm aversion. Journal of Marketing Research 56, 5 (2019), 809–825.

[9]

Joseph Cazier, Benjamin Shao, and Robert St Louis. 2017. Value congruence, trust, and their effects on purchase intention and reservation price. ACM Transactions on Management Information Systems 8, 4 (2017), 1–28.

Digital Library

[10]

Bogeum Choi, Austin Ward, Yuan Li, Jaime Arguello, and Robert Capra. 2019. The effects of task complexity on the use of different types of information in a search assistance tool. ACM Transactions on Information Systems 38, 1 (2019), 1–28.

Digital Library

[11]

Robyn M. Dawes. 1971. A case study of graduate admissions: Application of three principles of human decision making.American Psychologist 26, 2 (1971), 180.

[12]

Robyn M. Dawes. 1979. The robust beauty of improper linear models in decision making.American Psychologist 34, 7 (1979), 571.

[13]

Maria De-Arteaga, Riccardo Fogliato, and Alexandra Chouldechova. 2020. A case for humans-in-the-loop: Decisions in the presence of erroneous algorithmic scores. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–12.

[14]

Berkeley J. Dietvorst and Soaham Bharti. 2020. People reject algorithms in uncertain decision domains because they have diminishing sensitivity to forecasting error. Psychological Science 31, 10 (2020), 1302–1314.

[15]

Berkeley J. Dietvorst, Joseph P. Simmons, and Cade Massey. 2015. Algorithm aversion: People erroneously avoid algorithms after seeing them err.Journal of Experimental Psychology: General 144, 1 (2015), 114.

[16]

Berkeley J. Dietvorst, Joseph P. Simmons, and Cade Massey. 2018. Overcoming algorithm aversion: People will use imperfect algorithms if they can (even slightly) modify them. Management Science 64, 3 (2018), 1155–1170.

Digital Library

[17]

Robert Fildes and Paul Goodwin. 2007. Against your better judgment? How organizations can improve their use of management judgment in forecasting. Interfaces 37, 6 (2007), 570–576.

Digital Library

[18]

Ben Green and Yiling Chen. 2019. The principles and limits of algorithm-in-the-loop decision making. Proceedings of the ACM on Human-Computer Interaction 3, CSCW (2019), 1–24.

Digital Library

[19]

William M. Grove, David H. Zald, Boyd S. Lebow, Beth E. Snitz, and Chad Nelson. 2000. Clinical versus mechanical prediction: A meta-analysis.Psychological Assessment 12, 1 (2000), 19.

[20]

Junius Gunaratne, Lior Zalmanson, and Oded Nov. 2018. The persuasive power of algorithmic and crowdsourced advice. Journal of Management Information Systems 35, 4 (2018), 1092–1120.

[21]

Kotaro Hara, Abigail Adams, Kristy Milland, Saiph Savage, Chris Callison-Burch, and Jeffrey P. Bigham. 2018. A data-driven analysis of workers’ earnings on Amazon Mechanical Turk. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1–14.

Digital Library

[22]

Michael P. Haselhuhn, Maurice E. Schweitzer, and Alison M. Wood. 2010. How implicit beliefs influence trust recovery. Psychological Science 21, 5 (2010), 645–648.

[23]

Kevin Anthony Hoff and Masooda Bashir. 2015. Trust in automation: Integrating empirical evidence on factors that influence trust. Human Factors 57, 3 (2015), 407–434.

[24]

Panagiotis G. Ipeirotis. 2010. Analyzing the amazon mechanical turk marketplace. XRDS: Crossroads, The ACM Magazine for Students 17, 2 (2010), 16–21.

Digital Library

[25]

Leanna Ireland. 2020. Who errs? Algorithm aversion, the source of judicial error, and public support for self-help behaviors. Journal of Crime and Justice 43, 2 (2020), 174–192.

[26]

Ryan Kennedy, Philip Waggoner, and Matthew Ward. 2022. Trust in public policy algorithms. The Journal of Politics 84, 2 (2022), 1132–1148.

[27]

Taemie Kim and Pamela Hinds. 2006. Who should I blame? Effects of autonomy and transparency on attributions in human-robot interaction. In Proceedings of the 15th IEEE International Symposium on Robot and Human Interactive Communication. IEEE, 80–85.

[28]

Daniël Lakens, Anne M. Scheel, and Peder M. Isager. 2018. Equivalence testing for psychological research: A tutorial. Advances in Methods and Practices in Psychological Science 1, 2 (2018), 259–269.

[29]

Hyung Koo Lee, Jong Seok Lee, and Mark Keil. 2018. Using perspective-taking to de-escalate launch date commitment for products with known software defects. Journal of Management Information Systems 35, 4 (2018), 1251–1276.

[30]

John Lee and Neville Moray. 1992. Trust, control strategies and allocation of function in human-machine systems. Ergonomics 35, 10 (1992), 1243–1270.

[31]

John D. Lee and Neville Moray. 1994. Trust, self-confidence, and operators’ adaptation to automation. International Journal of Human-Computer Studies 40, 1 (1994), 153–184.

Digital Library

[32]

John D. Lee and Katrina A. See. 2004. Trust in automation: Designing for appropriate reliance. Human Factors 46, 1 (2004), 50–80.

[33]

Qing Li, Yuanzhu Chen, Li Ling Jiang, Ping Li, and Hsinchun Chen. 2016. A tensor-based information framework for predicting the stock market. ACM Transactions on Information Systems 34, 2 (2016), 1–30.

Digital Library

[34]

Kai H. Lim, Izak Benbasat, and Lawrence M. Ward. 2000. The role of multimedia in changing first impression bias. Information Systems Research 11, 2 (2000), 115–136.

Digital Library

[35]

Yu-Kai Lin, Hsinchun Chen, Randall A. Brown, Shu-Hsing Li, and Hung-Jen Yang. 2017. Healthcare predictive analytics for risk profiling in chronic care: A Bayesian multitask learning approach.MIS Quarterly 41, 2 (2017), 473–495.

Digital Library

[36]

Jennifer M. Logg, Julia A. Minson, and Don A. Moore. 2019. Algorithm appreciation: People prefer algorithmic to human judgment. Organizational Behavior and Human Decision Processes 151, C (2019), 90–103.

[37]

Chiara Longoni, Andrea Bonezzi, and Carey K. Morewedge. 2019. Resistance to medical artificial intelligence. Journal of Consumer Research 46, 4 (2019), 629–650.

[38]

Robert B. Lount, Chen-Bo Zhong, Niro Sivanathan, and J. Keith Murnighan. 2008. Getting off on the wrong foot: The timing of a breach and the restoration of trust. Personality and Social Psychology Bulletin 34, 12 (2008), 1601–1612.

[39]

Xueming Luo, Siliang Tong, Zheng Fang, and Zhe Qu. 2019. Frontiers: Machines vs. humans: The impact of artificial intelligence chatbot disclosure on customer purchases. Marketing Science 38, 6 (2019), 937–947.

Digital Library

[40]

Dietrich Manzey, Juliane Reichenbach, and Linda Onnasch. 2012. Human performance consequences of automated decision aids: The impact of degree of automation and system experience. Journal of Cognitive Engineering and Decision Making 6, 1 (2012), 57–87.

[41]

Scott Mayer McKinney, Marcin Sieniek, Varun Godbole, Jonathan Godwin, Natasha Antropova, Hutan Ashrafian, Trevor Back, Mary Chesus, Greg S. Corrado, Ara Darzi, Mozziyar Etemadi, Florencia Garcia-Vicente, Fiona J. Gilbert, Mark Halling-Brown, Demis Hassabis, Demis Sunny Jansen, Alan Karthikesalingam, Christopher J. Kelly, Dominic King, Joseph R. Ledsam, David Melnick, Hormuz Mostofi, Lily Peng, Joshua Jay Reicher, Bernardino Romera-Paredes, Richard Sidebottom, Mustafa Suleyman, Daniel Tse, Kenneth C. Young, Jeffrey De Fauw, and Shravya Shetty. 2020. International evaluation of an AI system for breast cancer screening. Nature 577, 7788 (2020), 89–94.

[42]

D. Harrison Mcknight, Michelle Carter, Jason Bennett Thatcher, and Paul F. Clay. 2011. Trust in a specific technology: An investigation of its components and measures. ACM Transactions on Management Information Systems 2, 2 (2011), 1–25.

Digital Library

[43]

Mahsan Nourani, Donald R. Honeycutt, Jeremy E. Block, Chiradeep Roy, Tahrima Rahman, Eric D. Ragan, and Vibhav Gogate. 2020. Investigating the importance of first impressions and explainable AI with interactive video analysis. In Proceedings of the Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems. 1–8.

Digital Library

[44]

Mahsan Nourani, Joanie King, and Eric Ragan. 2020. The role of domain expertise in user trust and the impact of first impressions with intelligent systems. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, Vol. 8. 112–121.

[45]

Paul A. Pavlou and Angelika Dimoka. 2006. The nature and role of feedback text comments in online marketplaces: Implications for trust building, price premiums, and seller differentiation. Information Systems Research 17, 4 (2006), 392–414.

Digital Library

[46]

Andrew Prahl and Lyn Van Swol. 2017. Understanding algorithm aversion: When is advice from automation discounted?Journal of Forecasting 36, 6 (2017), 691–702.

[47]

Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. “Why should i trust you?” Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 1135–1144.

Digital Library

[48]

Richard M. Ryan and Edward L. Deci. 2000. Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being.American Psychologist 55, 1 (2000), 68.

[49]

Julian Sanchez. 2006. Factors that Affect Trust and Reliance on an Automated Aid. Georgia Institute of Technology.

[50]

Nada R. Sanders and Karl B. Manrodt. 2003. The efficacy of using judgmental versus quantitative forecasting methods in practice. Omega 31, 6 (2003), 511–522.

[51]

James Schaffer, John O’Donovan, James Michaelis, Adrienne Raglin, and Tobias Höllerer. 2019. I can do better than your AI: Expertise and explanations. In Proceedings of the 24th International Conference on Intelligent User Interfaces. 240–251.

Digital Library

[52]

Oliver Schilke, Martin Reimann, and Karen S. Cook. 2013. Effect of relationship experience on trust recovery following a breach. Proceedings of the National Academy of Sciences 110, 38 (2013), 15236–15241.

[53]

Robert P. Schumaker and Hsinchun Chen. 2009. Textual analysis of stock market prediction using breaking financial news: The AZFin text system. ACM Transactions on Information Systems 27, 2 (2009), 1–19.

Digital Library

[54]

Maurice E. Schweitzer and Gérard P. Cachon. 2000. Decision bias in the newsvendor problem with a known demand distribution: Experimental evidence. Management Science 46, 3 (2000), 404–420.

[55]

Donghui Shi, Jian Guan, Jozef Zurada, and Andrew Manikas. 2017. A data-mining approach to identification of risk factors in safety management systems. Journal of Management Information Systems 34, 4 (2017), 1054–1081.

[56]

John J. Skowronski and Donal E. Carlston. 1989. Negativity and extremity biases in impression formation: A review of explanations.Psychological Bulletin 105, 1 (1989), 131.

[57]

Scott I. Vrieze and William M. Grove. 2009. Survey on the use of clinical and mechanical prediction methods in clinical psychology.Professional Psychology: Research and Practice 40, 5 (2009), 525.

[58]

Weiquan Wang and Izak Benbasat. 2008. Attributions of trust in decision support technologies: A study of recommendation agents for e-commerce. Journal of Management Information Systems 24, 4 (2008), 249–273.

Digital Library

[59]

Weiquan Wang and Izak Benbasat. 2016. Empirical assessment of alternative designs for enhancing different types of trusting beliefs in online recommendation agents. Journal of Management Information Systems 33, 3 (2016), 744–775.

[60]

Jenna Wiens and Erica S. Shenoy. 2018. Machine learning for healthcare: On the verge of a major shift in healthcare epidemiology. Clinical Infectious Diseases 66, 1 (2018), 149–153.

[61]

Michael Yeomans, Anuj Shah, Sendhil Mullainathan, and Jon Kleinberg. 2019. Making sense of recommendations. Journal of Behavioral Decision Making 32, 4 (2019), 403–414.

[62]

Kun Yu, Shlomo Berkovsky, Dan Conway, Ronnie Taib, Jianlong Zhou, and Fang Chen. 2016. Trust and reliance based on system accuracy. In Proceedings of the 2016 Conference on User Modeling Adaptation and Personalization. 223–227.

Digital Library

[63]

Kun Yu, Shlomo Berkovsky, Ronnie Taib, Dan Conway, Jianlong Zhou, and Fang Chen. 2017. User trust dynamics: An investigation driven by differences in system performance. In Proceedings of the 22nd International Conference on Intelligent User Interfaces. 307–317.

Digital Library

Cited By

Schoeffer JDe-Arteaga MKühl N(2024)Explanations, Fairness, and Appropriate Reliance in Human-AI Decision-MakingProceedings of the CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642621(1-18)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642621
Ha SMonadjemi SOttley A(2024)Guided By AI: Navigating Trust, Bias, and Data Exploration in AI‐Guided Visual AnalyticsComputer Graphics Forum10.1111/cgf.1510843:3Online publication date: 10-Jun-2024
https://doi.org/10.1111/cgf.15108
Kim MLee KBalan RLee Y(2023)Bubbleu: Exploring Augmented Reality Game Design with Uncertain AI-based InteractionProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581270(1-18)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581270

Index Terms

When Algorithms Err: Differential Impact of Early vs. Late Errors on Users’ Reliance on Algorithms
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Empirical studies in HCI
2. Information systems
  1. Information systems applications
    1. Decision support systems
      1. Data analytics

Recommendations

Impact of errors in operational spreadsheets

All users of spreadsheets struggle with the problem of errors. Errors are thought to be prevalent in spreadsheets, and in some instances they have cost organizations millions of dollars. In a previous study of 50 operational spreadsheets we found errors ...
To err is human, not algorithmic – Robust reactions to erring algorithms
Abstract
When seeing algorithms err, we trust them less and decrease using them compared to after seeing humans err; this is called algorithm aversion. This paper builds on the algorithm aversion literature and the third-party reactions to ...
Highlights
- Reactions toward erring algorithms go beyond algorithm aversion.
- Gut reactions ...
On the impact of hybrid errors on mobile WiMAX networks

Burst and AWGN errors affect mobile WiMAX networks because of propagation conditions generally resulting from the mobility characteristics. Mobile WiMAX networks are inherently vulnerable to transmission errors due to propagation conditions such as ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Computer-Human Interaction

ACM Transactions on Computer-Human Interaction Volume 30, Issue 1

February 2023

537 pages

ISSN:1073-0516

EISSN:1557-7325

DOI:10.1145/3585399

Editors:
Kristina Höök
KTH Royal Institute of Technology, Sweden
,
Kasper Hornbæk
University of Copenhagen, Denmark

Issue’s Table of Contents

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 March 2023

Online AM: 17 August 2022

Accepted: 12 July 2022

Revised: 20 June 2022

Received: 19 May 2021

Published in TOCHI Volume 30, Issue 1

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
824
Total Downloads

Downloads (Last 12 months)447
Downloads (Last 6 weeks)30

Reflects downloads up to 03 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Schoeffer JDe-Arteaga MKühl N(2024)Explanations, Fairness, and Appropriate Reliance in Human-AI Decision-MakingProceedings of the CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642621(1-18)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642621
Ha SMonadjemi SOttley A(2024)Guided By AI: Navigating Trust, Bias, and Data Exploration in AI‐Guided Visual AnalyticsComputer Graphics Forum10.1111/cgf.1510843:3Online publication date: 10-Jun-2024
https://doi.org/10.1111/cgf.15108
Kim MLee KBalan RLee Y(2023)Bubbleu: Exploring Augmented Reality Game Design with Uncertain AI-based InteractionProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581270(1-18)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581270

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Full Text

View this article in Full Text.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View full text|Download PDF

View Issue’s Table of Contents