research-article

Open access

AI governance in the system development life cycle: insights on responsible machine learning engineering

Authors:

Teemu Birkstedt,

Matti Mäantymäki,

Matti Minkkinen,

Tommi MikkonenAuthors Info & Claims

CAIN '22: Proceedings of the 1st International Conference on AI Engineering: Software Engineering for AI

Pages 113 - 123

https://doi.org/10.1145/3522664.3528598

Published: 17 October 2022 Publication History

Abstract

In this study we explore the incorporation of artificial intelligence (AI) governance to system development life cycle (SDLC) models. We conducted expert interviews among AI and SDLC professionals and analyzed the interview data using qualitative coding and clustering to extract AI governance concepts. Subsequently, we mapped these concepts onto three stages in the machine learning (ML) system development process: (1) design, (2) development, and (3) operation. We discovered 20 governance concepts, some of which are relevant to more than one of the three stages. Our analysis highlights AI governance as a complex process that involves multiple activities and stakeholders. As development projects are unique, the governance requirements and processes also vary. This study is a step towards understanding how AI governance is conceptually connected to ML systems' management processes through the project life cycle.

References

[1]

Rama Akkiraju, Vibha Sinha, Anbang Xu, Jalal Mahmud, Pritam Gundecha, Zhe Liu, Xiaotong Liu, and John Schumacher. 2020. Characterizing Machine Learning Processes: A Maturity Framework. In International Conference on Business Process Management. Springer, 17--31.

[2]

Alejandro Barredo Arrieta, Natalia Díaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador García, Sergio Gil-López, Daniel Molina, Richard Benjamins, et al. 2020. Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Information Fusion 58 (2020), 82--115.

Digital Library

[3]

Aleksandre Asatiani, Pekka Malo, Per Rådberg Nagbøl, Esko Penttinen, Tapani Rinta-Kahila, and Antti Salovaara. 2021. Sociotechnical Envelopment of Artificial Intelligence: An Approach to Organizational Deployment of Inscrutable Artificial Intelligence Systems. Journal of the Association for Information Systems 22, 2 (2021), 8.

[4]

S Balaji and M Sundararajan Murugaiyan. 2012. Waterfall vs. V-Model vs. Agile: A comparative study on SDLC. International Journal of Information Technology and Business Management 2, 1 (2012), 26--30.

[5]

Hind Benbya, Stella Pachidi, and Sirkka Jarvenpaa. 2021. Special Issue Editorial: Artificial Intelligence in Organizations: Implications for Information Systems Research. Journal of the Association for Information Systems 22, 2 (2021), 10.

[6]

James Butcher and Irakli Beridze. 2019. What is the State of Artificial Intelligence Governance Globally? The RUSI Journal 164, 5-6 (2019), 88--96.

[7]

Corinne Cath. 2018. Governing artificial intelligence: ethical, legal and technical opportunities and challenges.

[8]

LINDSEY Coombes, Debby Allen, Deborah Humphrey, and JOANNE Neale. 2009. In-depth interviews. Research methods for health and social care (2009), 197--210.

[9]

Allan Dafoe. 2018. AI governance: a research agenda. Governance of AI Program, Future of Humanity Institute, University of Oxford: Oxford, UK (2018).

[10]

Filip Karlo Došilović, Mario Brčić, and Nikica Hlupić. 2018. Explainable artificial intelligence: A survey. In 2018 41st International convention on information and communication technology, electronics and microelectronics (MIPRO). IEEE, 0210--0215.

[11]

Christof Ebert, Gorka Gallardo, Josune Hernantes, and Nicolas Serrano. 2016. DevOps. Ieee Software 33, 3 (2016), 94--100.

Digital Library

[12]

Ray Eitel-Porter. 2021. Beyond the promise: implementing ethical AI. AI and Ethics 1, 1 (2021), 73--80.

[13]

European Commission. 2021. Proposal for a Regulation Laying down Harmonised Rules on Artificial Intelligence (Artificial Intelligence Act) | Shaping Europe's Digital Future. https://digital-strategy.ec.europa.eu/en/library/proposal-regulation-laying-down-harmonised-rules-artificial-intelligence-artificial-intelligence. (2021).

[14]

Dror G Feitelson, Eitan Frachtenberg, and Kent L Beck. 2013. Development and deployment at Facebook. IEEE Internet Computing 17, 4 (2013), 8--17.

Digital Library

[15]

Urs Gasser and Virgilio AF Almeida. 2017. A layered model for AI governance. IEEE Internet Computing 21, 6 (2017), 58--62.

[16]

Bill Gillham. 2005. Research Interviewing: The range of techniques: A practical guide. McGraw-Hill Education (UK).

[17]

Dennis A Gioia, Kevin G Corley, and Aimee L Hamilton. 2013. Seeking qualitative rigor in inductive research: Notes on the Gioia methodology. Organizational research methods 16, 1 (2013), 15--31.

[18]

Fuyuki Ishikawa and Nobukazu Yoshioka. 2019. How do engineers perceive difficulties in engineering of machine-learning systems?-questionnaire survey. In 2019 IEEE/ACM Joint 7th International Workshop on Conducting Empirical Studies in Industry (CESI) and 6th International Workshop on Software Engineering Research and Industrial Practice (SER&IP). IEEE, 2--9.

[19]

Norman P Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, et al. 2017. In-datacenter performance analysis of a tensor processing unit. In Proceedings of the 44th annual international symposium on computer architecture. 1--12.

Digital Library

[20]

Stephan Jüngling, Martin Peraic, and Andreas Martin. 2020. Towards AI-based Solutions in the System Development Lifecycle. In AAAI Spring Symposium: Combining Machine Learning with Knowledge Engineering (1).

[21]

Ioannis Karamitsos, Saeed Albarhami, and Charalampos Apostolopoulos. 2020. Applying DevOps practices of continuous automation for machine learning. Information 11, 7 (2020), 363.

[22]

Nate Keating. 2020. An introduction to MLOps on Google Cloud. In available at: https://www.youtube.com/watch?v=6gdrwFMaEZ0, accessed 5 February 2021. Google.

[23]

Vijay Khatri and Carol V Brown. 2010. Designing data governance. Commun. ACM 53, 1 (2010), 148--152.

Digital Library

[24]

Srinivas Kudaravalli, Samer Faraj, and Steven L Johnson. 2017. A Configural Approach to Coordinating Expertise in Software Development Teams. MIS quarterly 41, 1 (2017).

[25]

Samuli Laato, Ali Farooq, Henri Tenhunen, Tinja Pitkamaki, Antti Hakkala, and AnttiAirola.2020. AI in Cybersecurity Education-A Systematic Literature Review of Studies on Cybersecurity MOOCs. In 2020 IEEE 20th International Conference on Advanced Learning Technologies (ICALT). IEEE, 6--10.

[26]

Samuli Laato, Matti Mäntymäki, Sami Hyrynsalmi, AKM Islam, and Teemu Birkstedt. 2022. Trends and Trajectories in the Software Industry: Implications for the Future of Work. Information System Frontiers (2022).

[27]

Samuli Laato, Henna Vilppu, Juho Heimonen, Antti Hakkala, Jari Björne, Ali Farooq, Tapio Salakoski, and Antti Airola. 2020. Propagating AI Knowledge Across University Disciplines-The Design of A Multidisciplinary AI Study Module. In 2020 IEEE Frontiers in Education Conference (FIE). IEEE, 1--9.

[28]

Harold A Linstone, Murray Turoff, et al. 1975. The delphi method. Addison-Wesley Reading, MA.

[29]

Jamal Mahboob and Joel Coffman. 2021. A Kubernetes CI/CD Pipeline with Asylo as a Trusted Execution Environment Abstraction Framework. In 2021 IEEE 11th Annual Computing and Communication Workshop and Conference (CCWC). IEEE, 0529--0535.

[30]

Matti Mäntymäki, Sami Hyrynsalmi, and Antti Koskenvoima. 2019. How do small and medium-sized game companies use analytics? An attention-based view of game analytics. Information Systems Frontiers (2019), 1ä16.

[31]

Matti Mäntymäki, Matti Minkkinen, Teemu Birkstedt, and Mika Viljanen. 2022. Defining organizational AI governance. AI and Ethics (2022), 1--7.

[32]

Matti Minkkinen, Anniina Niukkanen Zimmer, and Matti Mäntymäki. 2022. What about investors? ESG analyses as tools for ethics-based AI auditing. AI and Society (2022).

Digital Library

[33]

Matti Minkkinen, Markus Philipp Zimmer, and Matti Mantymaki. 2021. Towards Ecosystems for Responsible AI. In Conference on e-Business, e-Services and e-Society. Springer, 220--232.

[34]

Jessica Morley, Luciano Floridi, Libby Kinsey, and Anat Elhalal. 2020. From What to How: An Initial Review of Publicly Available AI Ethics Tools, Methods and Research to Translate Principles into Practices. Science and Engineering Ethics 26, 4 (Aug. 2020), 2141--2168.

[35]

Shyam Varan Nath. 2007. Champion-challenger based predictive model selection. In Proceedings 2007 IEEE SoutheastCon. IEEE, 254--254.

[36]

OECD. 2019. Recommendation of the Council on Artificial Intelligence, OECD/LEGAL/0449. (2019).

[37]

Alexander Poth, Mark Werner, and Xinyan Lei. 2018. How to Deliver Faster with CI/CD Integrated Testing Services?. In European Conference on Software Process Improvement. Springer, 401--409.

[38]

Iyad Rahwan. 2018. Society-in-the-Loop: Programming the Algorithmic Social Contract. Ethics and Information Technology 20, 1 (March 2018), 5--14.

Digital Library

[39]

Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. Model-agnostic interpretability of machine learning. arXiv preprint arXiv:1606.05386 (2016).

[40]

Martin L Rubin. 1970. Introduction to the system life cycle. Handbook of Data Processing Management. Vol. 1. (1970).

[41]

Johannes Schneider, Rene Abraham, and Christian Meske. 2020. AI Governance for Businesses. arXiv preprint arXiv:2011.10672 (2020).

[42]

Akseli Seppälä, Teemu Birkstedt, and Matti Mantymaki. 2021. From Ethical AI Principles to Governed AI. Proceedings of the 42nd International Conference on Information Systems (ICIS2021) (2021).

[43]

Ben Shneiderman. 2020. Bridging the Gap Between Ethics and Practice: Guidelines for Reliable, Safe, and Trustworthy Human-Centered AI Systems. Acm Transactions on Interactive Intelligent Systems 10, 4 (Dec. 2020), 26.

Digital Library

[44]

Victoria Stodden. 2020. The data science life cycle: a disciplined approach to advancing data science as a science. Commun. ACM 63, 7 (2020), 58--66.

Digital Library

[45]

Anselm Strauss and Juliet Corbin. 1998. Basics of qualitative research techniques. Citeseer.

[46]

R Andrew Taylor, Joseph R Pare, Arjun K Venkatesh, Hani Mowafi, Edward R Melnick, William Fleischman, and M Kennedy Hall. 2016. Prediction of in-hospital mortality in emergency department patients with sepsis: a local big data-driven, machine learning approach. Academic emergency medicine 23, 3 (2016), 269--278.

[47]

Ville Vakkuri, Kai-Kristian Kemell, and Pekka Abrahamsson. 2020. ECCOLA-a method for implementing ethically aligned AI systems. In 2020 46th Euromicro Conference on Software Engineering and Advanced Applications (SEAA). IEEE, 195--204.

[48]

Valohai. 2021. Practical MLOps: How to Get Ready for Production Models. ebook [ONLINE], available at: https://valohai.com/mlops-ebook/, accessed 18th of January.

[49]

Wil MP Van der Aalst. 2014. Data scientist: The engineer of the future. In Enterprise interoperability VI. Springer, 13--26.

[50]

Michael Veale, Reuben Binns, and Max Van Kleek. 2018. Some HCI priorities for GDPR-compliant machine learning. arXiv preprint arXiv:1803.06174 (2018).

[51]

Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michael Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. 2019. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature 575, 7782 (2019), 350--354.

[52]

Manish Virmani. 2015. Understanding DevOps & bridging the gap from continuous integration to continuous delivery. In Fifth international conference on the innovative computing technology (intech 2015). IEEE, 78--82.

[53]

Jie M Zhang, Mark Harman, Lei Ma, and Yang Liu. 2020. Machine learning testing: Survey, landscapes and horizons. IEEE Transactions on Software Engineering (2020).

Digital Library

Cited By

Watson HLarson D(2024)MLOpsInternational Journal of Business Intelligence Research10.4018/IJBIR.35891615:1(1-22)Online publication date: 1-Nov-2024
https://doi.org/10.4018/IJBIR.358916
Ogrizović MDrašković DBojić D(2024)Quality assurance strategies for machine learning applications in big data analytics: an overviewJournal of Big Data10.1186/s40537-024-01028-y11:1Online publication date: 30-Oct-2024
https://doi.org/10.1186/s40537-024-01028-y
Wu JBosch JLewis GCleland-Huang JMuccini H(2024)An Exploratory Study of V-Model in Building ML-Enabled Software: A Systems Engineering PerspectiveProceedings of the IEEE/ACM 3rd International Conference on AI Engineering - Software Engineering for AI10.1145/3644815.3644951(30-40)Online publication date: 11-Jun-2024
https://doi.org/10.1145/3644815.3644951
Show More Cited By

Index Terms

AI governance in the system development life cycle: insights on responsible machine learning engineering
1. Software and its engineering
  1. Software creation and management

Recommendations

Introducing an Iterative Life-Cycle Model at Credit Suisse IT Switzerland

Credit Suisse IT Switzerland decided to introduce an iterative life-cycle model (ILCM) based on the Rational Unified Process (RUP), a de facto industry standard, in addition to the well-established waterfall life-cycle model (WLCM). Such a large-scale ...
A Systematic Literature Review on Global Software Development Life Cycle

Global software development (GSD) has now become a prominent software development paradigm. Software companies are increasingly adopting GSD approaches in order to produce high quality software. GSD's popularity has attracted the researchers to ...
Selection of appropriate software development life cycle using fuzzy logic

Selection of appropriate Software Development Life Cycles can increase projects success. Depending on the selected SDLC Software Development Life Cycle, one can decrease development time/cost, minimize overhead and risk exposure, manage uncertainty, ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CAIN '22: Proceedings of the 1st International Conference on AI Engineering: Software Engineering for AI

May 2022

254 pages

ISBN:9781450392754

DOI:10.1145/3522664

General Chair:
Ivica Crnkovic
Chalmers University of Technology, SE

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

In-Cooperation

IEEE TCSC: IEEE Technical Committee on Scalable Computing

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 October 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

CAIN '22

Sponsor:

SIGSOFT

CAIN '22: 1st Conference on AI Engineering - Software Engineering for AI

May 16 - 24, 2022

Pennsylvania, Pittsburgh

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

11
Total Citations
View Citations
1,856
Total Downloads

Downloads (Last 12 months)1,045
Downloads (Last 6 weeks)137

Reflects downloads up to 25 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Watson HLarson D(2024)MLOpsInternational Journal of Business Intelligence Research10.4018/IJBIR.35891615:1(1-22)Online publication date: 1-Nov-2024
https://doi.org/10.4018/IJBIR.358916
Ogrizović MDrašković DBojić D(2024)Quality assurance strategies for machine learning applications in big data analytics: an overviewJournal of Big Data10.1186/s40537-024-01028-y11:1Online publication date: 30-Oct-2024
https://doi.org/10.1186/s40537-024-01028-y
Wu JBosch JLewis GCleland-Huang JMuccini H(2024)An Exploratory Study of V-Model in Building ML-Enabled Software: A Systems Engineering PerspectiveProceedings of the IEEE/ACM 3rd International Conference on AI Engineering - Software Engineering for AI10.1145/3644815.3644951(30-40)Online publication date: 11-Jun-2024
https://doi.org/10.1145/3644815.3644951
Billeter YDenzel PChavarriaga RForster OSchilling FBrunner SFrischknecht-Gruber CReif MWeng J(2024)MLOps as Enabler of Trustworthy AI2024 11th IEEE Swiss Conference on Data Science (SDS)10.1109/SDS60720.2024.00013(37-40)Online publication date: 30-May-2024
https://doi.org/10.1109/SDS60720.2024.00013
Donia JOyefeso LEmbuldeniya GWhyne CBurns DBoyer PRazmjou HShaw J(2024)Lifecycles, pipelines, and value chains: toward a focus on events in responsible artificial intelligence for healthAI and Ethics10.1007/s43681-024-00594-4Online publication date: 22-Oct-2024
https://doi.org/10.1007/s43681-024-00594-4
Leuthe DMeyer-Hollatz TPlank TSenkmüller A(2024)Towards Sustainability of AI – Identifying Design Patterns for Sustainable Machine Learning DevelopmentInformation Systems Frontiers10.1007/s10796-024-10526-6Online publication date: 16-Sep-2024
https://doi.org/10.1007/s10796-024-10526-6
Khattak AZhang JChan PChen FAlmujibah H(2023)Explainable Boosting Machine: A Contemporary Glass-Box Strategy for the Assessment of Wind Shear Severity in the Runway Vicinity Based on the Doppler Light Detection and Ranging DataAtmosphere10.3390/atmos1501002015:1(20)Online publication date: 23-Dec-2023
https://doi.org/10.3390/atmos15010020
Minkkinen MMäntymäki M(2023)Discerning Between the “Easy” and “Hard” Problems of AI GovernanceIEEE Transactions on Technology and Society10.1109/TTS.2023.32673824:2(188-194)Online publication date: Jun-2023
https://doi.org/10.1109/TTS.2023.3267382
Ma'ady MHidayat AAnaking PKusumawati ANasrullah MIstyanto NAsfari U(2023)Making Sense of Developing Artificial Intelligence-Based System in Software Development Life Cycle Manner and Addressing Risk Factors2023 6th International Conference of Computer and Informatics Engineering (IC2IE)10.1109/IC2IE60547.2023.10331070(244-249)Online publication date: 14-Sep-2023
https://doi.org/10.1109/IC2IE60547.2023.10331070
Nahar NZhang HLewis GZhou SKästner C(2023)A Meta-Summary of Challenges in Building Products with ML Components – Collecting Experiences from 4758+ Practitioners2023 IEEE/ACM 2nd International Conference on AI Engineering – Software Engineering for AI (CAIN)10.1109/CAIN58948.2023.00034(171-183)Online publication date: May-2023
https://doi.org/10.1109/CAIN58948.2023.00034
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents