Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
research-article
Open access

Conversational Agents for Information Retrieval in the Education Domain: A User-Centered Design Investigation

Published: 11 November 2022 Publication History
  • Get Citation Alerts
  • Abstract

    Text-based conversational agents (CAs) are widely deployed across a number of daily tasks, including information retrieval. However, most existing agents follow a default design that disregards user needs and preferences, ultimately leading to a lack of usage and an unsatisfying user experience. To better understand how CAs can be designed in order to lead to effective system use, we deduced relevant design requirements from both literature and 13 user interviews. We built and tested a question-answering, text-based CA for an information retrieval task in an education scenario. Results from our experimental test with 41 students indicate that following a user-centered design has a significant positive effect on enjoyment and trust in a CA as opposed to deploying a default CA. If not designed with the user in mind, CAs are not necessarily more beneficial than traditional question-answering systems. Beyond practical implications for effective CA design, this paper points towards key challenges and potential research avenues when deploying social cues for CAs.

    Supplementary Material

    MP4 File (v6cscw2486.mp4)
    Supplemental video

    References

    [1]
    Ritu Agarwal and Elena Karahanna. 2000. Time Flies When You're Having Fun: Cognitive Absorption and Beliefs about Information Technology Usage. MIS Quarterly 24, 4 (12 2000), 665. https://doi.org/10.2307/3250951
    [2]
    Mutaz M Al-Debei. 2014. The quality and acceptance of websites: An empirical investigation in the context of higher education. International Journal of Business Information Systems 15, 2 (2014), 170--188. https://doi.org/10.1504/IJBIS. 2014.059252
    [3]
    T. Araujo. 2018. Living up to the chatbot hype: The influence of anthropomorphic design cues and communicative agency framing on conversational agent and company perceptions. Computers in Human Behavior 85 (2018), 183--189.
    [4]
    Z. Ashktorab, M. Jain, V. Q. Liao, and J. D. Weisz. 2019. Perceptions of chatbots in therapy. In Proceedings of the ACM CHI Conference on Human Factors in Computing Systems. Glasgow, Scotland.
    [5]
    Z. Ashktorab, M. Jain, V. Q. Liao, and J. D. Weisz. 2019. Resilient chatbots: Repair strategy preferences for conversational breakdowns. In Proceedings of the ACM CHI Conference on Human Factors in Computing Systems. Glasgow, Scotland.
    [6]
    Nicholas Belkin. 1993. Interaction with Texts: Information Retrieval as Information-Seeking Behavior. (1993), 55--66. http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.50.6725
    [7]
    Benedikt Berger, Martin Adam, Alexander Rühr, and Alexander Benlian. 2021. Watch Me Improve-Algorithm Aversion and Demonstrating the Ability to Learn. Business and Information Systems Engineering 63, 1 (2021), 55--68. https://doi.org/10.1007/s12599-020-00678--5
    [8]
    Christopher G. Brinton, Ruediger Rill, Sangtae Ha, Mung Chiang, Robert Smith, and William Ju. 2015. Individualization for education at Scale: MIIC design and preliminary evaluation. IEEE Transactions on Learning Technologies 8, 1 (3 2015), 136--148. https://doi.org/10.1109/TLT.2014.2370635
    [9]
    J. Cambre and C. Kulkarni. 2019. One voice fits all? Social implications and research challenges of designing voices for smart devices. In Proceedings of the ACM on Human-Computer Interaction, 3(CSCW).
    [10]
    Sofy Carayannopoulos. 2018. Using chatbots to aid transition. International Journal of Information and Learning Technology 35, 2 (2018), 118--129. https://doi.org/10.1108/IJILT-10--2017-0097
    [11]
    R. C. S. Chang, H. P. Lu, and P. Yang. 2018. Stereotypes or golden rules? Exploring likable voice traits of social robots as active aging companions for tech-savvy baby boomers in Taiwan. Computers in Human Behavior 84 (2018), 194--210. https://doi.org/10.1016/j.chb.2018.02.025
    [12]
    Sue Inn Ch'ng, Lee Seng Yeong, and Xin Yean Ang. 2019. Preliminary Findings of using Chat-bots as a Course FAQ Tool. In 2019 IEEE Conference on e-Learning, e-Management and e-Services, IC3e 2019. Institute of Electrical and Electronics Engineers Inc., 52--56. https://doi.org/10.1109/IC3e47558.2019.8971786
    [13]
    Mike Cohn. 2004. User Stories Applied For Agile Software Development. Technical Report.
    [14]
    Harris M. Cooper. 1988. Organizing knowledge syntheses: A taxonomy of literature reviews. Knowledge in Society 1, 1 (1988), 104--126. https://doi.org/10.1007/BF03177550
    [15]
    Chris Creed and Russell Beale. 2008. Emotional Intelligence: Giving Computers Effective Emotional Skills to Aid Interaction. In Computational Intelligence: A Compendium.
    [16]
    Stephan Diederich, Alfred Benedikt Brendel, and Lutz M. Kolbe. 2020. Designing Anthropomorphic and Communicative Enterprise Conversational Agents. Business & Information Systems Engineering 4 (2020). https://doi.org/10.1007/s12599- 020-00639-y
    [17]
    Stephan Diederich, Alfred Benedikt Brendel, Stefan Morana, and Lutz Kolbe. 2022. On the Design of and Interaction with Conversational Agents: An Organizing and Assessing Review of Human-Computer Interaction Research. JAIS 23 (2022), 96--138. https://doi.org/10.17705/1jais.00724
    [18]
    Aaron C Elkins and Douglas C Derrick. 2013. The Sound of Trust: Voice as a Measurement of Trust During Interactions with Embodied Conversational Agents. Group Decision and Negotiation 22, 5 (2013), 897--913. https://doi.org/10.1007/ s10726-012--9339-x
    [19]
    Sean B. Eom, H. Joseph Wen, and Nicholas Ashill. 2006. The Determinants of Students' Perceived Learning Outcomes and Satisfaction in University Online Education: An Empirical Investigation*. Decision Sciences Journal of Innovative Education 4, 2 (7 2006), 215--235. https://doi.org/10.1111/j.1540--4609.2006.00114.x
    [20]
    Jasper Feine, Ulrich Gnewuch, Stefan Morana, and Alexander Maedche. 2019. A Taxonomy of Social Cues for Conversational Agents. International Journal of Human Computer Studies 132 (12 2019), 138--161. https://doi.org/10. 1016/j.ijhcs.2019.07.009
    [21]
    L. K. Fryer, M. Ainley, A. Thompson, A. Gibson, and Z. Sherlock. 2017. Stimulating and sustaining interest in a language course: An experimental comparison of Chatbot and Human task partners. Computers in Human Behavior 75 (2017), 461--46.
    [22]
    A. Følstad and P. B. Brandtzæg. 2017. Chatbots and the new world of HCI. Interactions 24, 4 (2017), 38--42. https: //doi.org/10.1016/j.chb.2018.02.025
    [23]
    David Gefen and Detmar W Straub. 2004. Consumer trust in B2C e-Commerce and the importance of social presence: Experiments in e-Products and e-Services. Omega 32, 6 (2004), 407--424. https://doi.org/10.1016/j.omega.2004.01.006
    [24]
    Justin Scott Giboney, Susan A. Brown, Paul Benjamin Lowry, and Jay F. Nunamaker. 2015. User acceptance of knowledge-based system recommendations: Explanations, arguments, and fit. Decision Support Systems 72 (4 2015), 1--10. https://doi.org/10.1016/j.dss.2015.02.005
    [25]
    J. Glaeser and G. Laude. 2010. Experteninterviews und qualitative Inhaltsanalyse : als Instrumente rekonstruierender Untersuchungen. VS Verlag fuer Sozialwissenschaften. https://link.springer.com/book/9783531172385
    [26]
    Jochen. Glaser and Grit. Laudel. 2010. Experteninterviews und qualitative Inhaltsanalyse : als Instrumente rekonstruierender Untersuchungen. VS Verlag fur Sozialwiss. http://www.springer.com/de/book/9783531172385
    [27]
    U Gnewuch, S Morana, M Adam, and A Maedche. 2017. This is the author ' s version of a work that was published in the following source Please note : Copyright is owned by the author and / or the publisher . Commercial use is not allowed . Institute of Information Systems and Marketing ( IISM ) The psychop. Thirty Eighth International Conference on Information Systems, South Korea 2017 December (2017), 0--11.
    [28]
    U. Gnewuch, S. Morana, M. T . P . Adam, and A. Maedche. 2018. Faster Is Not Always Better: Understanding the Effect of Dynamic Response Delays in Human-Chatbot Interaction. In Proceedings of the European Conference on Information Systems (ECIS). Portsmouth, United Kingdom.
    [29]
    Nicola Guarino and Christopher A Welty. 2004. An overview of OntoClean. In Handbook on ontologies. Springer, 151--171.
    [30]
    Paul Hemp. 2009. Death by information overload. Harvard business review 87, 9 (2009), 82--9.
    [31]
    Soomin Kim, Joonhwan Lee, and Gahgene Gweon. 2019. Comparing data from chatbot and web surveys effects of platform and conversational style on survey response quality. Conference on Human Factors in Computing Systems - Proceedings (2019), 1--12. https://doi.org/10.1145/3290605.3300316
    [32]
    Bernhard Kratzwald and Stefan Feuerriegel. 2019. Putting Question-Answering Systems into Practice. ACM Transactions on Management Information Systems 9, 4 (2019), 1--20. https://doi.org/10.1145/3309706
    [33]
    S. Laumer, A Racheva, F. Gubler, and C. Maier. 2019. Use Cases for Conversational Agents : An Interview-based Study. In Proceedings of the Americas Conference on Information Systems (AMCIS). Cancun, Mexico.
    [34]
    Mori. M. 1970. The Uncanny Valley. Energy 7, 4 (1970), 33--35.
    [35]
    Philipp Mayring. 2000. Qualitative content analysis. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research [On-line Journal] 2, 1 (2000).
    [36]
    D Harrison McKnight, Peng Liu, and Brian T Pentland. 2020. Trust Change in Information Technology Products. Journal of Management Information Systems 37, 4 (2020), 1015--1046. https://doi.org/10.1080/07421222.2020.1831772
    [37]
    P. Meier, J. H. Beinke, C. Fitte, and F. Behne, A.and Teuteberg. 2019. FeelFit -- Design and Evaluation of a Conversational Agent to Enhance Health Awareness. In Proceedings of the International Conference on Information Systems (ICIS).
    [38]
    Siddhant Meshram, Namit Naik, Megha VT, Tanmay More, and Shubhangi Kharche. 2021. College Enquiry Chatbot using Rasa Framework. In Asian Conference on Innovation in Technology (ASIANCON). Pune, India.
    [39]
    Fernando A. Mikic-Fonte, Martin Llamas-Nistal, and Manuel Caeiro-Rodriguez. 2019. Using a Chatterbot as a FAQ Assistant in a Course about Computers Architecture. Proceedings - Frontiers in Education Conference, FIE 2018-Octob (2019), 2018--2021. https://doi.org/10.1109/FIE.2018.8659174
    [40]
    Yi Mou and Kun Xu. 2017. The media inequality: Comparing the initial human-human and human-AI social interactions. Computers in Human Behavior 72, 2017 (2017), 432--440. https://doi.org/10.1016/j.chb.2017.02.067
    [41]
    Sergazy Narynov, Zhandos Zhumanov, Aidana Gumar, Mariyam Khassanova, and Matyrkhan Omarov. 2021. Development of Chatbot Psychologist Applying Natural Language Understanding Techniques. In Twentyfirst International Conference on Control, Automation and Systems (ICCAS 2021).
    [42]
    Clifford Nass and Youngme Moon. 2000. Machines and Mindlessness: Social Responses to Computers. Journal of Social Issues 56, 1 (2000), 81--103. https://doi.org/10.1111/0022--4537.00153
    [43]
    Clifford Nass, Jonathan Steuer, and Ellen R. Tauber. 1994. Computers are social actors. In Proceedings of the SIGCHI conference on Human factors in computing systems celebrating interdependence - CHI '94. ACM Press, New York, New York, USA, 72--78. https://doi.org/10.1145/191666.191703
    [44]
    Implementation Of, A Chatbot In, Online Higher, and Education Settings. 2018. Issues in Information Systems DESIGN AND IMPLEMENTATION OF A CHATBOT IN ONLINE HIGHER Issues in Information Systems. 19, 4 (2018), 44--52.
    [45]
    R. Othman. 2006. The Quality Indicators for an Information Retrieval System: User's Perspective. In 2006 2nd International Conference on Information Communication Technologies, Vol. 1. 1738--1744. https://doi.org/10.1109/ICTTA. 2006.1684648
    [46]
    Nicolas Pfeuffer, Alexander Benlian, Henner Gimpel, and Oliver Hinz. 2019. Anthropomorphic Information Systems. Business & Information Systems Engineering 61, 4 (2019), 523--533. https://doi.org/10.1007/s12599-019-00599-y
    [47]
    Aleksandra Przegalinska, Leon Ciechanowski, Anna Stroz, Peter Gloor, and Grzegorz Mazurek. 2019. In bot we trust: A new methodology of chatbot performance measures. Business Horizons 62, 6 (2019), 785--797. https://doi.org/10. 1016/j.bushor.2019.08.005
    [48]
    P. Puranam and B.S. Vanneste. 2021. Artificial Intelligence, Trust, and Perceptions of Agency. Working Paper (2021). https://doi.org/abstract=3897704
    [49]
    Dragomir Radev, Weiguo Fan, Hong Qi, Harris Wu, and Amardeep Grewal. 2005. Probabilistic question answering on the Web. Journal of the American Society for Information Science and Technology 56, 6 (4 2005), 571--583. https: //doi.org/10.1002/asi.20146
    [50]
    Bhavika R. Ranoliya, Nidhi Raghuwanshi, and Sanjay Singh. 2017. Chatbot for university related FAQs. 2017 International Conference on Advances in Computing, Communications and Informatics, ICACCI 2017 2017-Janua (2017), 1525--1530. https://doi.org/10.1109/ICACCI.2017.8126057
    [51]
    John T.E. Richardson. 2005. Instruments for obtaining student feedback: A review of the literature. Assessment and Evaluation in Higher Education 30, 4 (2005), 387--415. https://doi.org/10.1080/02602930500099193
    [52]
    Eric Ries. 2011. The Lean Startup: How Today's Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses.
    [53]
    Dmitri Roussinov and José A Robles-Flores. 2007. Applying question answering technology to locating malevolent online content. Decision Support Systems 43, 4 (2007), 1404--1418. https://doi.org/10.1016/j.dss.2006.04.006
    [54]
    Sherry Ruan, Liwei Jiang, Justin Xu, Bryce Joe-Kun Tham, Zhengneng Qiu, Yeshuang Zhu, Elizabeth L. Murnane, Emma Brunskill, and James A. Landay. 2019. QuizBot: A Dialogue-based Adaptive Learning System System for Factual Knowledge. Chi (2019), 1--13. https://doi.org/10.1145/3290605.3300587
    [55]
    Victoria L. Rubin, Yimin Chen, and Lynne Marie Thorimbert. 2010. Artificially intelligent conversational agents in libraries. Library Hi Tech 28, 4 (2010), 496--522. https://doi.org/10.1108/07378831011096196
    [56]
    I. Scarpellini and Y. Lim. 2020. Role-Based Design of Conversational Agents: Approach and Tools. In HCI International 2020 - Late Breaking Posters.
    [57]
    Anuschka Schmitt, Wambsganss, and Andreas Janson. 2022. Designing for Conversational System Trustworthiness: The Impact of Model Transparency on Trust and Task Performance. In Thirtieth European Conference on Information Systems. Timisoara, Romania, 1--18.
    [58]
    Anuschka Schmitt, Thiemo Wambsganss, Matthias Soellner, and Andreas Janson. 2021. Towards a Trust Reliance Paradox? Exploring the Gap Between Perceived Trust in and Reliance on Algorithmic Advice. In Forty-Second International Conference on Information Systems. Austin, Texas, USA, 1--17.
    [59]
    A.-M. Seeger, J. Pfeiffer, and A. Heinzl. 2018. Designing Anthropomorphic Conversational Agents: Development and Empirical Evaluation of a Design Framework. In Proceedings of the International Conference on Information Systems (ICIS). San Francisco, USA.
    [60]
    John V. Seidel. 1998. Qualitative data analysis. http://eer.engin.umich.edu/wp-content/uploads/sites/443/2019/08/SeidelQualitative-Data-Analysis.pdf
    [61]
    Bayan Abu Shawar and Eric Steven Atwell. 2005. Using corpora in machine-learning chatbot systems. International Journal of Corpus Linguistics 10, 4 (2005), 489--516. https://doi.org/10.1075/ijcl.10.4.06sha
    [62]
    Nicole Shechtman and Leonard M Horowitz. 2003. Media inequality in conversation. 5 (2003), 281. https://doi.org/10. 1145/642659.642661
    [63]
    R. F. Simmons. 1965. Answering English questions by computer: A survey. Commun. ACM 8, 1 (1 1965), 53--70. https://doi.org/10.1145/363707.363732
    [64]
    Ronnie W Smith. [n.d.]. Integration of domain problem solving with natural language dialog : the missing axiom theory. 1 ([n. d.]).
    [65]
    Pavel Smutny and Petra Schreiberova. 2020. Chatbots for learning: A review of educational chatbots for the Facebook Messenger. Computers and Education 151, June 2019 (2020), 103862. https://doi.org/10.1016/j.compedu.2020.103862
    [66]
    Eriks Sneiders. 1999. Automated FAQ Answering: Continued Experience with Shallow Language Understanding. Question Answering Systems. Papers from the 1999 AAAI Fall Symposium Bubenko 1994 (1999), 97--107. http://www. aaai.org/Papers/Symposia/Fall/1999/FS-99-02/FS99-02-017.pdf
    [67]
    Elliot Soloway, Mark Guzdial, and Kenneth E Hay. 1994. Learner-Centered Design The Challenge For WC1 In The Xst Century. Interactions (1994), 36--48.
    [68]
    Yasunobu Sumikawa, Masaaki Fujiyoshi, Hisashi Hatakeyama, and Masahiro Nagai. 2019. Supporting creation of FAQ dataset for e-learning Chatbot. Smart Innovation, Systems and Technologies 142, February (2019), 3--13. https://doi.org/10.1007/978--981--13--8311--3{_}1
    [69]
    Ilaria Torre, Jeremy Goslin, and Laurence White. 2020. If your device could smile: People trust happy-sounding artificial agents more. Computers in Human Behavior 105, December 2019 (2020), 106215. https://doi.org/10.1016/j.chb.2019.106215
    [70]
    Gabriele Trovato, Alexander Lopez, Renato Paredes, and Francisco Cuellar. 2017. Security and guidance: Two roles for a humanoid robot in an interaction experiment. RO-MAN 2017 - 26th IEEE International Symposium on Robot and Human Interactive Communication 2017-Janua (2017), 230--235. https://doi.org/10.1109/ROMAN.2017.8172307
    [71]
    Shahper Vodanovich, David Sundaram, and Michael Myers. 2010. Digital natives and ubiquitous information systems. Information Systems Research 21, 4 (2010), 711--723. https://doi.org/10.1287/isre.1100.0324
    [72]
    Jan vom Brocke, Alexander Simons, Kai Riemer, Bjoern Niehaves, Ralf Plattfaut, and Anne Cleven. 2015. Standing on the shoulders of giants: Challenges and recommendations of literature search in information systems research. Communications of the Association for Information Systems 37, 1 (8 2015), 205--224. https://doi.org/10.17705/1cais.03709
    [73]
    Raphael von Wolff, Jonas Nörtemann, Sebastian Hobert, and Matthias Schumann. 2020. Chatbots for the Information Acquisition at Universities -- A Student's View on the Application Area. In International Workshop on Chatbot Research and Design, Vol. 11970 LNCS. Springer, 231--244. https://doi.org/10.1007/978--3-030--39540--7{\{}{\T1\textbackslash}{_}{\}}16
    [74]
    Thiemo Wambsganß, Leonie Haas, and Matthias Söllner. 2021. Towards the Design of a Student-Centered QuestionAnswering System in Educational Settings. In Twenty-Ninth European Conference on Information Systems (ECIS 2021). Virtual Conference, 1--8.
    [75]
    Thiemo Wambsganss, Tobias Kueng, Matthias Soellner, and Jan Marco Leimeister. 2021. ArgueTutor: An Adaptive Dialog-Based Learning System for Argumentation Skills. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3411764. 3445781
    [76]
    Liz C Wang, Julie Baker, Judy A Wagner, and Kirk Wakefield. 2007. Can a retail Web Site be social? Journal of Marketing 71, 3 (2007), 143--157. https://doi.org/10.1509/jmkg.71.3.143
    [77]
    Florian Weber, Thiemo Wambsganss, Dominic Rüttimann, and Matthias Söllner. 2021. Pedagogical Agents for Interactive Learning: A Taxonomy of Conversational Agents in Education. In Forty-Second International Conference on Information Systems. Austin, Texas, USA, 1--17.
    [78]
    Tom D Wilson. 1999. Models in information behaviour research. Journal of documentation (1999).
    [79]
    R. Winkler and M. Söllner. 2018. Unleashing the Potential of Chatbots in Education : A State-Of-The-Art Analysis . In : Academy of Management. Meeting, Annual Chicago, A O M (2018). https://www.alexandria.unisg.ch/254848/1/JML_ 699.pdf
    [80]
    Rainer Winkler and Matthias Söllner. 2018. Unleashing the Potential of Chatbots in Education: A State-Of-The-Art Analysis. Academy of Management Proceedings (2018). https://doi.org/10.5465/ambpp.2018.15903abstract

    Cited By

    View all
    • (2024)Designing the Conversational Agent: Asking Follow-up Questions for Information ElicitationProceedings of the ACM on Human-Computer Interaction10.1145/36373208:CSCW1(1-30)Online publication date: 26-Apr-2024
    • (2023)Investigating the Support Provided by Chatbots to Educational Institutions and Their Students: A Systematic Literature ReviewMultimodal Technologies and Interaction10.3390/mti71101037:11(103)Online publication date: 4-Nov-2023
    • (2023)Designing a Chatbot for Contemporary Education: A Systematic Literature ReviewInformation10.3390/info1409050314:9(503)Online publication date: 13-Sep-2023

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image Proceedings of the ACM on Human-Computer Interaction
    Proceedings of the ACM on Human-Computer Interaction  Volume 6, Issue CSCW2
    CSCW
    November 2022
    8205 pages
    EISSN:2573-0142
    DOI:10.1145/3571154
    Issue’s Table of Contents
    This work is licensed under a Creative Commons Attribution International 4.0 License.

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 11 November 2022
    Published in PACMHCI Volume 6, Issue CSCW2

    Check for updates

    Author Tags

    1. conversational agent
    2. enjoyment
    3. interaction design
    4. trust
    5. user-centered

    Qualifiers

    • Research-article

    Funding Sources

    • Swiss National Science Foundation

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)508
    • Downloads (Last 6 weeks)59
    Reflects downloads up to 11 Aug 2024

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Designing the Conversational Agent: Asking Follow-up Questions for Information ElicitationProceedings of the ACM on Human-Computer Interaction10.1145/36373208:CSCW1(1-30)Online publication date: 26-Apr-2024
    • (2023)Investigating the Support Provided by Chatbots to Educational Institutions and Their Students: A Systematic Literature ReviewMultimodal Technologies and Interaction10.3390/mti71101037:11(103)Online publication date: 4-Nov-2023
    • (2023)Designing a Chatbot for Contemporary Education: A Systematic Literature ReviewInformation10.3390/info1409050314:9(503)Online publication date: 13-Sep-2023

    View Options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Get Access

    Login options

    Full Access

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media