research-article

Open access

Human-in-the-loop AI in government: a case study

Authors:

Lanthao Benedikt,

Chaitanya Joshi,

Ruben Henstra-Hill,

Sharon HookAuthors Info & Claims

IUI '20: Proceedings of the 25th International Conference on Intelligent User Interfaces

Pages 488 - 497

https://doi.org/10.1145/3377325.3377489

Published: 17 March 2020 Publication History

Abstract

In this paper, we present a novel application where Human-Computer Interaction (HCI) meets Artificial Intelligence (AI) and discuss obstacles that need to be resolved on the long journey from research to production. Unlike academia and industries that have been at the forefront of automation for decades, government is a new player in the field, though an important one. We build systems that are used on a large scale, we collect data to inform policymakers. Using the example of the Household Budget Survey, we demonstrate how government agencies can apply Human-in-the-Loop AI to automate the production of official statistics. The aim is time and resource saving on repetitive, labour-intensive tasks which machines are good at, allowing humans to focus on value added tasks requiring flexibility and intelligence. One major challenge is the human factor. How will the users, who are accustomed to manual tasks, react to the complexity of AI? How should we design the interface to give them a good user experience? How do we measure success? Indeed, one key step towards production is to secure funding, which requires presenting potential success in a way that the stakeholder can understand. Stressing the importance of formulating problems from a practical business viewpoint, we hope to bridge the communication gap and help the research community reach out to more potential users and help solve more novel real-world problems.

References

[1]

The UK National Archives. 2017. General hints and tips for digitisation for business use. Guidance and Best Practice (Sept. 2017). https://www.nationalarchives.gov.uk/documents/information-management/hints-tips-digitisation-for-business-use.pdf

[2]

Gustavo Batista, Ronaldo Cristiano Prati, and Maria Carolina Monard. 2004. A study of the behavior of several methods for balancing machine learning training data. ACM SIGKDD Explorations Newsletter 6, 1 (June 2004), 20--29.

Digital Library

[3]

Philip Bille and Martin Farach Colton. 2008. Fast and Compact Regular Expression Matching. Theoretical Computer Science 409, 3 (Dec. 2008). http://arxiv.org/abs/cs/0509069

Digital Library

[4]

George A. Boyne. 1992. Local Government Structure and Performance: Lessons from America? Public Administration. 70, 3 (Sept. 1992), 333--357.

[5]

Matteo Brisinello, Ratko Grbic, Matija Pul, and Tihomir Andelic. 2017. Improving optical character recognition performance for low quality images. In Proceedings of the 2017 International Symposium ELMAR. IEEE.

[6]

Lars Buitinck, Gilles Louppe, Mathieu Blondel, Fabian Pedregosa, Andreas Mueller, Olivier Grisel, Vlad Niculae, Peter Prettenhofer, Alexandre Gramfort, Jaques Grobler, Robert Layton, Jake VanderPlas, Arnaud Joly, Brian Holt, and Gael Varoquaux. 2013. API design for machine learning software: experiences from the scikit-learn project. ECML PKDD Workshop: Languages for Data Mining and Machine Learning (June 2013), 108--122. https://scikit-learn.org/stable/_downloads/scikit-learn-docs.pdf

[7]

Toxtli Carlos, Monroy-Hernandez Andres, and Cranshaw Justin. 2018. Understanding Chatbot-mediated Task Management. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (May 2018).

Digital Library

[8]

Gavin Cawley and Nicola Talbot; 11(Jul). 2010. On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation. Journal of Machine Learning Research (July 2010), 2079--2107. http://www.jmlr.org/papers/v11/cawley10a.html

[9]

J. S. Cramer. 2004. The origin of logistic regression. Studies in History and Philosophy of Science Part C: Studies in History and Philosophy of Biological and Biomedical Sciences 4, 35 (Jan. 2004).

[10]

Wenyuan Dai, GuiRong XueQiang, and Yang Yong Yu. 2007. Trans-ferring naive Bayes classifiers for text classification. Proc. of Assoc. for the Adv. of Art. Int. (AAAI 07) (Aug. 2007). http://new.aaai.org/Papers/AAAI/2007/AAAI07-085.pdf

[11]

de Amorim Renato Cordeiro and Zampieri Marcos. 2013. Effective Spell Checking Methods Using Clustering Algorithms. Proceedings of the International Conference Recent Advances in Natural Language Processing (Sept. 2013), 172--178.

[12]

United Nations Statistics Division. 2018. COICOP Revision. https://unstats.un.org/unsd/class/revisions/coicop_revision.asp

[13]

Cranor Lorrie Faith. [n.d.]. A Framework for Reasoning About the Human in the Loop. Proceedings of the 1st Conference on Usability, Psychology, and Security ([n. d.]).

[14]

Pieter Fivez, Simon Suster, and Walter Daelemans. 2017. Unsupervised Context-Sensitive Spelling Correction of English and Dutch Clinical Free-Text with Word and Character N-Gram Embeddings. BioNLP abs/1710.07045, 3 (Aug. 2017). http://arxiv.org/abs/1710.07045

[15]

Peter Flach. 2019. Performance Evaluation in Machine Learning: The Good, the Bad, the Ugly, and the Way Forward. AAAI Press 2019 (Jan. 2019), 9808--9814.

Digital Library

[16]

Miriam GilEmail, Vicente Pelechano, Joan Fons, and Manoli Albert. 2016. Designing the Human in the Loop of Self-Adaptive Systems. International Conference on Ubiquitous Computing and Ambient Intelligence (Nov. 2016), 437--449.

[17]

Github. 2019. Tesseract Releases. https://github.com/tesseract-ocr/tesseract/releases

[18]

Keith Hartley. 2009. Value for money in defence: Strategic choices and efficiency savings. The Chartered Institute of Public Finance and Accountancy. Public Money 5, 4 (Jan. 2009), 33--38.

[19]

Brink Henrik, Richards Joseph, and Fetherolf Mark. 2016. Real-World Machine Learning. Manning Publications Co. 1sr Ed. (Jan. 2016). https://doi.org/1617291927,9781617291920

[20]

Tin Kam Ho. 1998. The Random Subspace Method for Constructing Decision Forests. IEEE Transactions on Pattern Analysis and Machine Intelligence 8, 20 (Aug. 1998).

Digital Library

[21]

Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long short-term memory. Neural Computation 9, 8 (Jan. 1997).

Digital Library

[22]

V. J. Hodge and J. Austin. 2003. A comparison of standard spell checking algorithms and a novel binary neural approach. IEEE Transactions on Knowledge and Data Engineering 15, 5 (Sept. 2003), 1073--1081.

[23]

Kononenko Igor and Bratko Ivan. 1991. Information-Based Evaluation Criterion for Classifier's Performance. Machine Learning 6, 1 (Jan. 1991).

Digital Library

[24]

Michael P. Jones and James H. Martin. 1997. Contextual Spelling Correction Using Latent Semantic Analysis. Fifth Conference on Applied Natural Language Processing (Jan. 1997), 166--173.

Digital Library

[25]

Sharp Laure. 1981. Respondent Burden: A First Measurement Effort. Öffentliche Meinung und sozialer Wandel / Public Opinion and Social Change (Jan. 1981), 194--208.

[26]

Machine Learning. 2019. Supervised learning. https://en.wikipedia.org/wiki/Supervised_learning

[27]

Sokolova Marina, Japkowicz Nathalie, and Szpakowicz Stan. 2006. Beyond Accuracy, F-Score and ROC: A Family of Discriminant Measures for Performance Evaluation. AI 2006: Advances in Artificial Intelligence (Jan. 2006), 1015--1021.

Digital Library

[28]

Banko Michele and Brill Eric. 2001. Mitigating the Paucity-of-data Problem: Exploring the Effect of Training Corpus Size on Classifier Performance for Natural Language Processing. Proceedings of the First International Conference on Human Language Technology Research (Jan. 2001), 1--5.

Digital Library

[29]

Didrik Nielsen. 2016. Tree Boosting With XGBoost - Why Does XGBoost Win Every Machine Learning Competition? Master Thesis. Norwegian University of Science and Technology (Jan. 2016). https://ntnuopen.ntnu.no/ntnu-xmlui/bitstream/handle/11250/2433761/16128_FULLTEXT.pdf?sequence=1

[30]

Donald A Norman. 1990. The design of everyday things. New York: Doubleday Publishing Group (Dec. 1990). http://www.nixdell.com/classes/HCI-and-Design-Spring-2017/The-Design-of-Everyday-Things-Revised-and-Expanded-Edition.pdf

Digital Library

[31]

Scikit-Learn Python package. 20196. Model evaluation: quantifying the quality of predictions. https://scikit-learn.org/stable/modules/model_evaluation.html

[32]

F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research 12 (2011), 2825--2830.

Digital Library

[33]

Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (June 2018). https://arxiv.org/pdf/1802.05365.pdf

[34]

Bojanowski Piotr, Grave Edouard, Joulin Armand, and Mikolov Tomas. 2017. Enriching Word Vectors with Subword Information. Transactions of the Association for Computational Linguistics 5, 20 (Aug. 2017).

[35]

Polikar R. 2012. Ensemble learning, Springer. https://link.springer.com/chapter/10.1007/978-1-4419-9326-7_1

[36]

Parasuraman R. and Sheridan Thomas. 2000. A model for types and levels of human interaction with automation. IEEE Transactions on Systems Man and Cybernetics - Part A Systems and Humans 30, 3 (Jan. 2000). http://hci.cs.uwaterloo.ca/faculty/elaw/cs889/reading/automation/sheridan.pdf

[37]

Nitin Ramesh, Aksha Srivastava, and K.Deeba. 2018. Improving Optical Character Recognition Techniques. International Journal of Engineering and Technology 7, 2.24 (Jan. 2018), 361--364.

[38]

Ling Rothrock and S. Narayanan. 2011. Human-in-the-Loop Simulations: Methods and Practice. Springer (Jan. 2011). https://doi.org/ISBN978-0-85729-883-6

[39]

Ingersoll Grant S., Thomas S. Morton, and Andrew L. Farris. 2013. Taming Text: How to Find, Organize, and Manipulate It. Manning Publications Co. (Jan. 2013). https://doi.org/193398838X,9781933988382

[40]

Aditya Siddhant, Anuj Goyal, and Angeliki Metallinou. 2019. Unsupervised Transfer Learning for Spoken Language Understanding in Intelligent Agents. Proceedings of the AAAI Conference on Artificial Intelligence 33 (July 2019).

Digital Library

[41]

Ray Smith. 2007. An Overview of the Tesseract OCR Engine. In Proceedings of the 9th International Conference on Document Analysis and Recognition. IEEE.

[42]

Ray Smith. 2016. Training LSTM on 100 languages and test results. https://github.com/tesseract-ocr/docs/blob/master/das_tutorial2016/7Building%20a%20Multi-Lingual%20OCR%20Engine.pdf

[43]

Rice Stephen, Frank Jenkins, and Thomas Nartker. 2013. The Fourth Annual Test of OCR Accuracy. http://www.expervision.com/wp-content/uploads/2012/12/1995.The_Fourth_Annual_Test_of_OCR_Accuracy.pdf

[44]

Yichuan Tang. 2013. Deep Learning using Support Vector Machines. CoRR (Aug. 2013). http://arxiv.org/abs/1306.0239

[45]

Li Wenchao, Sadigh Dorsa, Sastry S. Shankar, and Seshia Sanjit A. 2014. Synthesis for Human-in-the-Loop Control Systems. Tools and Algorithms for the Construction and Analysis of Systems 30, 3 (Jan. 2014), 470--484.

Cited By

Casado–Mansilla DGómez–Carmona OFernández–de–Retana MMuzzioli LKušar AVandevijvere SLópez–de–Ipiña D(2024)Food Assistant for Consumer Behaviour Change through Citizen Science and AI2024 9th International Conference on Smart and Sustainable Technologies (SpliTech)10.23919/SpliTech61897.2024.10612651(01-06)Online publication date: 25-Jun-2024
https://doi.org/10.23919/SpliTech61897.2024.10612651
Botella-Gil BSepúlveda-Torres RBonet-Jover AMartínez-Barco PSaquete E(2024)Semi-Automatic Dataset Annotation Applied to Automatic Violent Message DetectionIEEE Access10.1109/ACCESS.2024.336140412(19651-19664)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3361404
Schmager SGrøder CParmiggiani EPappas IVassilakopoulou P(2024)Exploring citizens’ stances on AI in public services: A social contract perspectiveData & Policy10.1017/dap.2024.136Online publication date: 22-Mar-2024
https://doi.org/10.1017/dap.2024.13
Show More Cited By

Index Terms

Human-in-the-loop AI in government: a case study
1. Computing methodologies
  1. Machine learning
    1. Machine learning approaches
2. Human-centered computing
  1. Human computer interaction (HCI)

Recommendations

Open government and e-government: democratic challenges from a public value perspective
dg.o '11: Proceedings of the 12th Annual International Digital Government Research Conference: Digital Government Innovation in Challenging Times

We consider open government (OG) within the context of e-government and its broader implications for the future of public administration. We argue that the current US Administration's Open Government Initiative blurs traditional distinctions between e-...
Using human-in-the-loop and explainable AI to envisage new future work practices
PETRA '22: Proceedings of the 15th International Conference on PErvasive Technologies Related to Assistive Environments

In this paper, we discuss the trends and challenges of the integration of Artificial Intelligence (AI) methods in the workplace. An important aspect towards creating positive AI futures in the workplace is the design of fair, reliable and trustworthy ...
Open government and e-government: democratic challenges from a public value perspective
Special issue on Open Government and Public Participation: Issues and Challenges in Creating Public Value

We argue that the Obama Administration's Open Government Initiative blurs distinctions between e-democracy and e-government by incorporating historically democratic practices. now enabled by emerging technology. within administrative agencies. We ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

IUI '20: Proceedings of the 25th International Conference on Intelligent User Interfaces

March 2020

607 pages

ISBN:9781450371186

DOI:10.1145/3377325

General Chairs:
Fabio Paternò,
Nuria Oliver,
Program Chairs:
Cristina Conati,
Lucio Davide Spano,
Nava Tintarev

Copyright © 2020 Owner/Author.

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 March 2020

Check for updates

Author Tags

Qualifiers

Research-article

Conference

IUI '20

Sponsor:

IUI '20: 25th International Conference on Intelligent User Interfaces

March 17 - 20, 2020

Cagliari, Italy

Acceptance Rates

Overall Acceptance Rate 746 of 2,811 submissions, 27%

Upcoming Conference

IUI '25

Sponsor:
sigai
sigai

30th International Conference on Intelligent User Interfaces

March 24 - 27, 2025

Cagliari , Italy

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
1,853
Total Downloads

Downloads (Last 12 months)573
Downloads (Last 6 weeks)89

Reflects downloads up to 16 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Casado–Mansilla DGómez–Carmona OFernández–de–Retana MMuzzioli LKušar AVandevijvere SLópez–de–Ipiña D(2024)Food Assistant for Consumer Behaviour Change through Citizen Science and AI2024 9th International Conference on Smart and Sustainable Technologies (SpliTech)10.23919/SpliTech61897.2024.10612651(01-06)Online publication date: 25-Jun-2024
https://doi.org/10.23919/SpliTech61897.2024.10612651
Botella-Gil BSepúlveda-Torres RBonet-Jover AMartínez-Barco PSaquete E(2024)Semi-Automatic Dataset Annotation Applied to Automatic Violent Message DetectionIEEE Access10.1109/ACCESS.2024.336140412(19651-19664)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3361404
Schmager SGrøder CParmiggiani EPappas IVassilakopoulou P(2024)Exploring citizens’ stances on AI in public services: A social contract perspectiveData & Policy10.1017/dap.2024.136Online publication date: 22-Mar-2024
https://doi.org/10.1017/dap.2024.13
Bonet-Jover ASepúlveda-Torres RSaquete EMartínez-Barco PPiad-Morffis AEstevez-Velarde S(2024)Applying Human-in-the-Loop to construct a dataset for determining content reliability to combat fake newsEngineering Applications of Artificial Intelligence10.1016/j.engappai.2023.107152126:PDOnline publication date: 27-Feb-2024
https://dl.acm.org/doi/10.1016/j.engappai.2023.107152
Papyshev G(2024)Governing AI through interaction: situated actions as an informal mechanism for AI regulationAI and Ethics10.1007/s43681-024-00446-1Online publication date: 27-Mar-2024
https://doi.org/10.1007/s43681-024-00446-1
Chen XWang XQu Y(2023)Constructing Ethical AI Based on the “Human-in-the-Loop” SystemSystems10.3390/systems1111054811:11(548)Online publication date: 13-Nov-2023
https://doi.org/10.3390/systems11110548
Jung HSeo WSong SNa S(2023)Toward Value Scenario Generation Through Large Language ModelsCompanion Publication of the 2023 Conference on Computer Supported Cooperative Work and Social Computing10.1145/3584931.3606960(212-220)Online publication date: 14-Oct-2023
https://dl.acm.org/doi/10.1145/3584931.3606960
Farhood HSaberi MNajafi M(2021)Human-in-the-Loop Optimization for Artificial Intelligence AlgorithmsService-Oriented Computing – ICSOC 2021 Workshops10.1007/978-3-031-14135-5_7(92-102)Online publication date: 22-Nov-2021
https://dl.acm.org/doi/10.1007/978-3-031-14135-5_7
Sohail M(2020)IoT Sensor Data Analysis and FusionEnabling AI Applications in Data Science10.1007/978-3-030-52067-0_17(381-396)Online publication date: 24-Sep-2020
https://doi.org/10.1007/978-3-030-52067-0_17

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents