research-article

A maturity assessment framework for conversational AI development platforms

Authors:

Johan Aronsson,

Daniel Strüber,

Thorsten BergerAuthors Info & Claims

SAC '21: Proceedings of the 36th Annual ACM Symposium on Applied Computing

Pages 1736 - 1745

https://doi.org/10.1145/3412841.3442046

Published: 22 April 2021 Publication History

Abstract

Conversational Artificial Intelligence (AI) systems have recently sky-rocketed in popularity and are now used in many applications, from car assistants to customer support. The development of conversational AI systems is supported by a large variety of software platforms, all with similar goals, but different focus points and functionalities. A systematic foundation for classifying conversational AI platforms is currently lacking. We propose a framework for assessing the maturity level of conversational AI development platforms. Our framework is based on a systematic literature review, in which we extracted common and distinguishing features of various open-source and commercial (or in-house) platforms. Inspired by language reference frameworks, we identify different maturity levels that a conversational AI development platform may exhibit in understanding and responding to user inputs. Our framework can guide organizations in selecting a conversational AI development platform according to their needs, as well as helping researchers and platform developers improving the maturity of their platforms.

References

[1]

2019. Iteragency Language Roundtable. (2019). https://www.govtilr.org/Skills/ILRscale2.htm

[2]

2019. Messenger Platform. (2019). https://developers.facebook.com/docs/messenger-platform/

[3]

Glenn Bowen. 2009. Document Analysis as a Qualitative Research Method. (2009).

[4]

Tom B. Brown and others. 2020. Language Models are Few-Shot Learners. In NeurIPS.

[5]

Massimo Canonico and Luigi De Russis. 2018. A Comparison and Critique of Natural Language Understanding Tools. In International Conference on Cloud Computing, GRIDs, and Virtualization. IARIA, Barcelona, 110--115.

[6]

Leigh Clark and others. 2019. What Makes a Good Conversation?: Challenges in Designing Truly Conversational Agents. In CHI Conference on Human Factors in Computing Systems.

[7]

Council of Europe. Common European Framework for Languages: Learning, teaching assessment. Technical Report. 21--42 pages. https://rm.coe.int/1680459f97

[8]

Krzysztof Czarnecki and Ulrich W. Eisenecker. 2000. Generative Programming: Methods, Tools, and Applications. Addison-Wesley.

Digital Library

[9]

Krzysztof Czarnecki and Simon Helsen. 2006. Feature-based survey of model transformation approaches. IBM Syst. J. 45 (July 2006), 621--645. Issue 3.

[10]

Sebastian Erdweg and others. 2013. The State of the Art in Language Workbenches. In International Conference on Software Language Engineering.

[11]

Jianfeng Gao, Michel Galley, and Lihong Li. 2018. Neural Approaches to Conversational AI Question Answering, Task-Oriented Dialogues and Social Chatbots. Technical Report. https://arxiv.org/pdf/1809.08267.pdf

[12]

Lan-Fen Huang, Simon Kubelec, Nicole Keng, and Lung-Hsun Hsu. 2018. Evaluating CEFR rater performance through the analysis of spoken learner corpora. (2018).

[13]

Kyo C Kang, Sholom G Cohen, James A Hess, William E Novak, and A Spencer Peterson. 1990. Feature-oriented domain analysis (FODA) feasibility study. Technical Report. Carnegie-Mellon University Pittsburgh.

[14]

Lorenz Cuno Klopfenstein, Saverio Delpriori, Silvia Malatini, and Alessandro Bogliolo. 2017. The Rise of Bots: A Survey of Conversational Interfaces, Patterns, and Paradigms. In Conference on Designing Interactive Systems, DIS. 555--565.

Digital Library

[15]

Dale L Lange and Jr Lowe, Pardee. 1987. Grading Reading Passages According to the ACTFL/ETS/ILR Reading Proficiency Standard: Can It Be Learned? Technical Report.

[16]

Kwanwoo Lee, Kyo Kang, and Jaejoon Lee. 2002. Concepts and Guidelines of Feature Modeling for Product Line Software Engineering. Technical Report. Pohang University of Science and Technology.

[17]

Iolanda Leite, André Pereira, Samuel Mascarenhas, Carlos Martinho, Rui Prada, and Ana Paiva. 2013. The influence of empathy in human-robot relations. International Journal of Human Computer Studies 71, 3 (2013), 250--260.

Digital Library

[18]

Conanico Massimo and Luigi De Russis. 2018. A Comparison and Critique of Natural Language Understanding Tools. In Cloud Computing 2018. Barcelona, 110--115.

[19]

Michael McTear. 2018. Conversational Modelling for Chatbots: Current Approaches and Future Directions. In Conference on Electronic Speech Signal Processing. 175--185.

[20]

Michael McTear, Zoraida Callejas, and David Griol. 2016. The Conversational Interface Talking to Smart Devices. Springer International Publishing. 51--68 pages.

[21]

Ram Menon. 2017. Council Post: The Rise Of Conversational AI. https://www.forbes.com/sites/forbestechcouncil/2017/12/04/the-rise-of-conversational-ai/#63d2a333b91b. (2017).

[22]

Eric Michiels. 2017. Modelling Chatbots with a Cognitive System Allows for a Differentiating User Experience. Technical Report. IBM, Belgium.

[23]

Damir Nešić, Jacob Krüger, Ştefan Stănciulescu, and Thorsten Berger. 2019. Principles of feature modeling. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. ACM, 62--73.

Digital Library

[24]

Amit Patil, K Marimuthu, Nagaraja Rao, and R Niranchana. 2017. Comparative study of cloud platforms to develop a Chatbot. International Journal of Engineering & Technology (2017), 57--61.

[25]

Ashwin Ram and others. 2017. Conversational AI: The Science Behind the Alexa Prize. Technical Report.

[26]

Bayan Abu Shawar and Eric Atwell. 2007. Different measurements metrics to evaluate a chatbot system. In Bridging the Gap: Academic and Industrial Research in Dialog Technologies Workshop Proceedings. Association for Computational Linguistics, Rochester, NY, 89--96.

Digital Library

[27]

Elvira Swender, J. Daniel Conrad, and Robert Vicars. 2012. ACTFL proficiency guidelines 2012. Technical Report. American Council on the Teaching of Foreign Languages, Alexandria. https://www.actfl.org/sites/default/files/pdfs/public/ACTFLProficiencyGuidelines2012_FINAL.pdf

[28]

The authors. 2020. Online appendix. (2020).

[29]

Peter Tsai. 2018. Data snapshot: AI Chatbots and Intelligent Assistants in the Workplace - Spiceworks. (2018). https://community.spiceworks.com/blog/2964-data-snapshot-ai-chatbots-and-intelligent-assistants-in-the-workplace

[30]

Erwing Tschierner, Olaf Bärenfänger, and Irmgard Wanner. 2012. Assessing Evidence of Validity of Assigning CEFR Ratings to the ACTFL Oral Proficiency Interview (OPI) and the Oral Proficiency Interview by computers (OPIc). Technical Report. Institute for Test Research and Development, Leipzig.

[31]

Anu Venkatesh and others. 2017. On Evaluating and Comparing Conversational Agents. In NIPS: Conversational AI Workshop.

[32]

Claes Wohlin. 2014. Guidelines for Snowballing in Systematic Literature Studies and a Replication in Software Engineering. In International Conference on Evaluation and Assessment in Software Engineering. 38:1--10.

Digital Library

[33]

Rui Yan. 2018. "Chitty-Chitty-Chat Bot": Deep Learning for Conversational AI. In IJCAI, Vol. 18. 5520--5526.

Cited By

de Wit JBraggaar A(2023)Tilbot: A Visual Design Platform to Facilitate Open Science Research into Conversational User InterfacesProceedings of the 5th International Conference on Conversational User Interfaces10.1145/3571884.3604403(1-5)Online publication date: 19-Jul-2023
https://dl.acm.org/doi/10.1145/3571884.3604403
Melo GNascimento NAlencar PCowan D(2023)Identifying Factors That Impact Levels of Automation in Autonomous SystemsIEEE Access10.1109/ACCESS.2023.328261711(56437-56452)Online publication date: 2023
https://doi.org/10.1109/ACCESS.2023.3282617
Ferner CLienbacher E(2023)Chatbots an physischen TouchpointsMarketingtechnologien10.1007/978-3-658-42294-3_12(159-172)Online publication date: 1-Dec-2023
https://doi.org/10.1007/978-3-658-42294-3_12
Show More Cited By

Index Terms

A maturity assessment framework for conversational AI development platforms
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction techniques

Recommendations

The appropriation of conversational AI in the workplace: A taxonomy of AI chatbot users
Abstract
There is an exponential growth of the use of AI applications in organisations. Due to the machine learning capability of artificial intelligence (AI) applications, it is critical that such systems are used continuously in order to ...
Highlights
- The study examines the use of AI chatbot or personal virtual assistant in the workplace.
The development of data analytics maturity assessment framework: DAMAF
Abstract
Today, data analytics plays a vital role in attaining competitive advantage, generating business value, and driving revenue streams for organizations. Thus, the organizations pay significant attention to improve their data analytics maturity. ...

This paper proposes the data analytics maturity assessment framework (DAMAF) to support organizations in assessing their current data analytics maturity, determining organizational gaps in data analytics, and proposing suggestions for data analytics ...
Security Maturity Self-Assessment Framework for Software Development Lifecycle
ARES '22: Proceedings of the 17th International Conference on Availability, Reliability and Security

Vulnerable software often originates from insufficient attention to security in the software development lifecycle. However, current maturity models provide limited support for the teams to assess the security maturity of their software development ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SAC '21: Proceedings of the 36th Annual ACM Symposium on Applied Computing

March 2021

2075 pages

ISBN:9781450381048

DOI:10.1145/3412841

Conference Chairs:
Chih-Cheng Hung
Kennesaw State University
,
Jiman Hong
Soongsil University, South Korea
,
Program Chairs:
Alessio Bechini
University of Pisa, Italy
,
Eunjee Song
Baylor University

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGAPP: ACM Special Interest Group on Applied Computing

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 22 April 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SAC '21

Sponsor:

SIGAPP

SAC '21: The 36th ACM/SIGAPP Symposium on Applied Computing

March 22 - 26, 2021

Virtual Event, Republic of Korea

Acceptance Rates

Overall Acceptance Rate 1,650 of 6,669 submissions, 25%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
408
Total Downloads

Downloads (Last 12 months)111
Downloads (Last 6 weeks)21

Reflects downloads up to 04 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

de Wit JBraggaar A(2023)Tilbot: A Visual Design Platform to Facilitate Open Science Research into Conversational User InterfacesProceedings of the 5th International Conference on Conversational User Interfaces10.1145/3571884.3604403(1-5)Online publication date: 19-Jul-2023
https://dl.acm.org/doi/10.1145/3571884.3604403
Melo GNascimento NAlencar PCowan D(2023)Identifying Factors That Impact Levels of Automation in Autonomous SystemsIEEE Access10.1109/ACCESS.2023.328261711(56437-56452)Online publication date: 2023
https://doi.org/10.1109/ACCESS.2023.3282617
Ferner CLienbacher E(2023)Chatbots an physischen TouchpointsMarketingtechnologien10.1007/978-3-658-42294-3_12(159-172)Online publication date: 1-Dec-2023
https://doi.org/10.1007/978-3-658-42294-3_12
Tarawneh ERousseau JCraig SChandola DKhan WFaizi AJenkin M(2023)An Infrastructure for Studying the Role of Sentiment in Human-Robot InteractionPattern Recognition, Computer Vision, and Image Processing. ICPR 2022 International Workshops and Challenges10.1007/978-3-031-37745-7_7(89-105)Online publication date: 29-Jul-2023
https://doi.org/10.1007/978-3-031-37745-7_7
Idowu SStrüber DBerger T(2022)Asset Management in Machine Learning: State-of-research and State-of-practiceACM Computing Surveys10.1145/354384755:7(1-35)Online publication date: 15-Dec-2022
https://dl.acm.org/doi/10.1145/3543847
Eltahir AAbdulla HPlatos JSnasel V(2022)Review of Chatbot Security Systems2022 26th International Conference on Circuits, Systems, Communications and Computers (CSCC)10.1109/CSCC55931.2022.00037(167-178)Online publication date: Jul-2022
https://doi.org/10.1109/CSCC55931.2022.00037
Idowu SStrüber DBerger TEldh SFalessi D(2021)Asset management in machine learningProceedings of the 43rd International Conference on Software Engineering: Software Engineering in Practice10.1109/ICSE-SEIP52600.2021.00014(51-60)Online publication date: 25-May-2021
https://dl.acm.org/doi/10.1109/ICSE-SEIP52600.2021.00014

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents