research-article

Will Algorithms Blind People? The Effect of Explainable AI and Decision-Makers’ Experience on AI-supported Decision-Making in Government

Authors:

Marijn Janssen,

Martijn Hartog,

Ricardo Matheus,

George KukAuthors Info & Claims

Social Science Computer Review, Volume 40, Issue 2

Pages 478 - 493

https://doi.org/10.1177/0894439320980118

Published: 01 April 2022 Publication History

Abstract

Computational artificial intelligence (AI) algorithms are increasingly used to support decision making by governments. Yet algorithms often remain opaque to the decision makers and devoid of clear explanations for the decisions made. In this study, we used an experimental approach to compare decision making in three situations: humans making decisions (1) without any support of algorithms, (2) supported by business rules (BR), and (3) supported by machine learning (ML). Participants were asked to make the correct decisions given various scenarios, while BR and ML algorithms could provide correct or incorrect suggestions to the decision maker. This enabled us to evaluate whether the participants were able to understand the limitations of BR and ML. The experiment shows that algorithms help decision makers to make more correct decisions. The findings suggest that explainable AI combined with experience helps them detect incorrect suggestions made by algorithms. However, even experienced persons were not able to identify all mistakes. Ensuring the ability to understand and traceback decisions are not sufficient for avoiding making incorrect decisions. The findings imply that algorithms should be adopted with care and that selecting the appropriate algorithms for supporting decisions and training of decision makers are key factors in increasing accountability and transparency.

References

[1]

Adadi A. and Berrada M. (2018). Peeking inside the black-box: A survey on Explainable Artificial Intelligence (XAI). IEEE Access, 6, 52138–52160.

[2]

Alpaydin E. (2009). Introduction to machine learning. MIT Press.

[3]

Arnaboldi M. (2018). The missing variable in big data for social sciences: The decision-maker. Sustainability, 10(10), 3415.

[4]

Barocas S. and Selbst A. D. (2016). Big data’s disparate impact. California Law Review, 104, 671.

[5]

Böcker A., Grütters C., Laemers M., Strik M., Terlouw A., and Zwaan K. (2014). Evaluatie van de herziene asielprocedure: Eindrapport [Evaluation of the revised asylum procedure: Final report]. Radboud University.

[6]

Brauneis R. and Goodman E. P. (2018). Algorithmic transparency for the smart city. Yale Journal of Law & Technology, 20, 103.

[7]

Brynjolfsson E. and Mitchell T. (2017). What can machine learning do? Workforce implications. Science, 358(6370), 1530–1534.

[8]

Burrell J. (2016). How the machine “thinks”: Understanding opacity in machine learning algorithms. Big Data & Society, 3(1). https://doi.org/10.1177/2053951715622512

[9]

Butler D. (2013). When Google got flu wrong. Nature, 494(7436), 155.

[10]

Churchman C. W. (1967). Free for all. Management Science, 14(4), 141–146.

Digital Library

[11]

de Sousa W. G., de Melo E. R. P., Bermejo P. H. D. S., Farias R. A. S., and Gomes A. O. (2019). How and where is artificial intelligence in the public sector going? A literature review and research agenda. Government Information Quarterly, 36(4), 101392. https://doi.org/10.1016/j.giq.2019.07.004

[12]

Diesner J. (2015). Small decisions with big impact on data analytics. Big Data & Society, 2(2), https://doi.org/10.1177/2053951715617185.

[13]

Duan Y., Edwards J. S., and Dwivedi Y. K. (2019). Artificial intelligence for decision making in the era of Big Data–evolution, challenges and research agenda. International Journal of Information Management, 48, 63–71.

Digital Library

[14]

Dwivedi Y. K., Rana N. P., Janssen M., Lal B., Williams M. D., and Clement M. (2017). An empirical validation of a unified model of electronic government adoption (UMEGA). Government Information Quarterly, 34(2), 211–230. https://doi.org/10.1016/j.giq.2017.03.001

[15]

Goertzel B. and Pennachin C. (2007). Artificial general intelligence (Vol. 2). Springer.

[16]

Gong Y. and Janssen M. (2012). From policy implementation to business process management: Principles for creating flexibility and agility. Government Information Quarterly, 29, S61–S71.

[17]

Gong Y. and Janssen M. (2013). An interoperable architecture and principles for implementing strategy and policy in operational processes. Computers in Industry, 64(8), 912–924.

Digital Library

[18]

Grace K., Salvatier J., Dafoe A., Zhang B., and Evans O. (2018). When will AI exceed human performance? Evidence from AI experts. Journal of Artificial Intelligence Research, 62, 729–754.

Digital Library

[19]

Höchtl J., Parycek P., and Schöllhammer R. (2016). Big data in the policy cycle: Policy decision making in the digital era. Journal of Organizational Computing and Electronic Commerce, 26(1–2), 147–169. https://doi.org/10.1080/10919392.2015.1125187

[20]

Hong S. and Lee S. (2018). Adaptive governance, status quo bias, and political competition: Why the sharing economy is welcome in some cities but not in others. Government Information Quarterly, 35(2), 283–290.

[21]

Hurley M. W. and Wallace W. A. (1986). Expert systems as decision aids for public managers: An assessment of the technology and prototyping as a design strategy. Public Administration Review, 46, 563–571.

[22]

Janssen M., Brous P., Estevez E., Barbosa L. S., and Janowski T. (2020). Data governance: Organizing data for trustworthy artificial intelligence. Government Information Quarterly, 37(4), 1–8. https://doi.org/10.1016/j.giq.2020.101493

[23]

Janssen M. and Kuk G. (2016a). Big and Open Linked Data (BOLD) in research, policy and practice. Journal of Organizational Computing and Electronic Commerce, 26(1–2), 3–13. https://doi.org/10.1080/10919392.2015.1124005

[24]

Janssen M. and Kuk G. (2016b). The challenges and limits of big data algorithms in technocratic governance. Government Information Quarterly, 33(3), 371–377. https://doi.org/10.1016/j.giq.2016.08.011

[25]

Jordan M. I. and Mitchell T. M. (2015). Machine learning: Trends, perspectives, and prospects. Science, 349(6245), 255–260.

[26]

Kankanhalli A., Charalabidis Y., and Mellouli S. (2019). IoT and AI for Smart Government: A research agenda. Government Information Quarterly, 36(2), 304–309. https://doi.org/10.1016/j.giq.2019.02.003

[27]

Kashin K., King G., and Soneji S. (2015). Explaining systematic bias and nontransparencyin US social security administration forecasts. Political Analysis, 23(3), 336–362. https://doi.org/10.1093/pan/mpv011

[28]

Kleinberg J., Lakkaraju H., Leskovec J., Ludwig J., and Mullainathan S. (2018). Human decisions and machine predictions. The Quarterly Journal of Economics, 133(1), 237–293.

[29]

Kroll J. A. (2015). Accountable algorithms. Doctoral Dissertation Princeton University. http://dataspace.princeton.edu/jspui/handle/88435/dsp014b29b837r

[30]

Lindblom C. E. (1959). The science of “muddling through.” Public Administration Review, 19, 79–88.

[31]

Miller T. (2019). Explanation in artificial intelligence: Insights from the social sciences. Artificial Intelligence, 267, 1–38. https://doi.org/10.1016/j.artint.2018.07.007

[32]

Raghunathan S. (1999). Impact of information quality and decision-maker quality on decision quality: A theoretical model and simulation analysis. Decision Support Systems, 26(4), 275–286. https://doi.org/10.1016/S0167-9236(99)00060-3

[33]

Rittel H. W. J. and Webber M. M. (1973). Dilemmas in a general theory of planning. Policy Sciences, 4(2), 155–169. https://doi.org/10.1007/BF01405730

[34]

Russell S. J. and Norvig P. (2016). Artificial intelligence: A modern approach. Pearson Education Limited.

Digital Library

[35]

Selbst A. D., Boyd D., Friedler S. A., Venkatasubramanian S., and Vertesi J. (2019). Fairness and abstraction in sociotechnical systems [Paper presentation]. Proceedings of the Conference on Fairness, Accountability, and Transparency. ACM. https://doi.org/10.1145/3287560.3287598

[36]

Simon H. A. (1972). Theories of bounded rationality. Decision and Organization, 1(1), 161–176.

[37]

Sun T. Q. and Medaglia R. (2019). Mapping the challenges of artificial intelligence in the public sector: Evidence from public healthcare. Government Information Quarterly, 36(2), 368–383. https://doi.org/10.1016/j.giq.2018.09.008

[38]

Turban E. and Watkins P. R. (1986). Integrating expert systems and decision support systems. MIS Quarterly, 10/2, 121–136.

[39]

Vanthienen J. (2001). Ruling the business: About business rules and decision tables. New Directions in Software Engineering, 103–120. https://www.semanticscholar.org/paper/Ruling-the-Business-%3A-About-Business-Rules-and-Vanthienen/8aaee7079dd06b75ad60e3a8535e5cf23752f1a3

[40]

Von Halle B. (2001). Business rules applied: Building better systems using the business rules approach. Wiley Publishing.

[41]

Weick K. E., Sutcliffe K. M., and Obstfeld D. (2005). Organizing and the process of sensemaking. Organization Science, 16(4), 409–421.

Digital Library

[42]

Zhang P., Zhao K., and Kumar R. L. (2016). Impact of IT governance and IT capability on firm performance. Information Systems Management, 33(4), 357–373.

Digital Library

Cited By

Nannini LHuyskes DPanai EPistilli GTartaro A(2025)Nullius in Explanans: an ethical risk assessment for explainable AIEthics and Information Technology10.1007/s10676-024-09800-727:1Online publication date: 1-Mar-2025
https://dl.acm.org/doi/10.1007/s10676-024-09800-7
Sampaio IFontes SAndrade EBernardini FViterbo J(2024)Enhancing Transparency through Explainable Artificial Intelligence: An Exploratory Analysis on Collusion and Corruption Scenario in Digital GovernmentProceedings of the 25th Annual International Conference on Digital Government Research10.1145/3657054.3657274(52-62)Online publication date: 11-Jun-2024
https://dl.acm.org/doi/10.1145/3657054.3657274
Eom SKim S(2024)The multi-dimensional paradox of AI adoption in the public sector: The Korean experienceProceedings of the 25th Annual International Conference on Digital Government Research10.1145/3657054.3657074(133-145)Online publication date: 11-Jun-2024
https://dl.acm.org/doi/10.1145/3657054.3657074
Show More Cited By

Index Terms

Will Algorithms Blind People? The Effect of Explainable AI and Decision-Makers’ Experience on AI-supported Decision-Making in Government

Index terms have been assigned to the content through auto-classification.

Recommendations

The design of AHPEC in web-based decision support system for making decision
AIC'10/BEBI'10: Proceedings of the 10th WSEAS international conference on applied informatics and communications, and 3rd WSEAS international conference on Biomedical electronics and biomedical informatics

This paper presents the architecture of the WDSS using AHPEC for making decision. The WDSS-AHPEC involves client, user interface, server, web server, database, and DSS model. The DSS model in this study is using AHPEC method. In applying AHPEC, it ...
Collaborative Decision Making

<P>A synthesis of the work of three noted authors provides a framework for collaborative decisions built on the foundation of decision analysis. A Nobel Prize winner provides a psychological foundation for the framework, an authority on harnessing the ...
Experience-Based Decision Making: A Satisficing Decision Tree Approach

This paper introduces a framework of experienced-based decision making as an extension of case-based decision making, a recently proposed alternative to expected utility theory. In experienced-based decision making, an agent faced with a new decision ...

Comments

Information & Contributors

Information

Published In

Social Science Computer Review Volume 40, Issue 2

Apr 2022

291 pages

ISSN:0894-4393

Issue’s Table of Contents

© The Author(s) 2020.

This article is distributed under the terms of the Creative Commons Attribution 4.0 License (https://creativecommons.org/licenses/by/4.0/) which permits any use, reproduction and distribution of the work without further permission provided the original work is attributed as specified on the SAGE and Open Access pages (https://us.sagepub.com/en-us/nam/open-access-at-sage).

Publisher

Sage Publications, Inc.

United States

Publication History

Published: 01 April 2022

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

12
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 08 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Nannini LHuyskes DPanai EPistilli GTartaro A(2025)Nullius in Explanans: an ethical risk assessment for explainable AIEthics and Information Technology10.1007/s10676-024-09800-727:1Online publication date: 1-Mar-2025
https://dl.acm.org/doi/10.1007/s10676-024-09800-7
Sampaio IFontes SAndrade EBernardini FViterbo J(2024)Enhancing Transparency through Explainable Artificial Intelligence: An Exploratory Analysis on Collusion and Corruption Scenario in Digital GovernmentProceedings of the 25th Annual International Conference on Digital Government Research10.1145/3657054.3657274(52-62)Online publication date: 11-Jun-2024
https://dl.acm.org/doi/10.1145/3657054.3657274
Eom SKim S(2024)The multi-dimensional paradox of AI adoption in the public sector: The Korean experienceProceedings of the 25th Annual International Conference on Digital Government Research10.1145/3657054.3657074(133-145)Online publication date: 11-Jun-2024
https://dl.acm.org/doi/10.1145/3657054.3657074
Kim DLee HLee SEom S(2024)Conditions for AI systems adoption in public sector: From an accountability perspectiveProceedings of the 25th Annual International Conference on Digital Government Research10.1145/3657054.3657060(42-51)Online publication date: 11-Jun-2024
https://dl.acm.org/doi/10.1145/3657054.3657060
Hupont IWainer MNester STissot SIglesias-Blanco LBaldassarri S(2024)Synocene, Beyond the Anthropocene: De-Anthropocentralising Human-Nature-AI InteractionExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3637118(1-9)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613905.3637118
Maragno GTangi LGastaldi LBenedetti M(2024)Exploring the factors, affordances and constraints outlining the implementation of Artificial Intelligence in public sector organizationsInternational Journal of Information Management: The Journal for Information Professionals10.1016/j.ijinfomgt.2023.10268673:COnline publication date: 5-Jan-2024
https://dl.acm.org/doi/10.1016/j.ijinfomgt.2023.102686
Wang SMin CLiang ZZhang YGao Q(2024)The decision-making by citizensComputers in Human Behavior10.1016/j.chb.2024.108413161:COnline publication date: 18-Nov-2024
https://dl.acm.org/doi/10.1016/j.chb.2024.108413
Rizk ALindgren I(2024)Automated Decision-Making in the Public Sector: A Multidisciplinary Literature ReviewElectronic Government10.1007/978-3-031-70274-7_15(237-253)Online publication date: 3-Sep-2024
https://dl.acm.org/doi/10.1007/978-3-031-70274-7_15
Delfos JZuiderwijk AVan Cranenburgh SChorus C(2022)Perceived challenges and opportunities of machine learning applications in governmental organisations: an interview-based exploration in the NetherlandsProceedings of the 15th International Conference on Theory and Practice of Electronic Governance10.1145/3560107.3560122(82-89)Online publication date: 4-Oct-2022
https://dl.acm.org/doi/10.1145/3560107.3560122
Medaglia RTangi L(2022)The adoption of Artificial Intelligence in the public sector in Europe: drivers, features, and impactsProceedings of the 15th International Conference on Theory and Practice of Electronic Governance10.1145/3560107.3560110(10-18)Online publication date: 4-Oct-2022
https://dl.acm.org/doi/10.1145/3560107.3560110
Show More Cited By

View Options

View options

Figures

Tables

Media

View Issue’s Table of Contents