research-article

Open access

Keep Me in the Loop: Increasing Operator Situation Awareness through a Conversational Multimodal Interface

Authors:

Francisco J. Chiyah Garcia,

Helen HastieAuthors Info & Claims

ICMI '18: Proceedings of the 20th ACM International Conference on Multimodal Interaction

Pages 384 - 392

https://doi.org/10.1145/3242969.3242974

Published: 02 October 2018 Publication History

Abstract

Autonomous systems are designed to carry out activities in remote, hazardous environments without the need for operators to micro-manage them. It is, however, essential that operators maintain situation awareness in order to monitor vehicle status and handle unforeseen circumstances that may affect their intended behaviour, such as a change in the environment. We present MIRIAM, a multimodal interface that combines visual indicators of status with a conversational agent component. This multimodal interface offers a fluid and natural way for operators to gain information on vehicle status and faults, mission progress and to set reminders. We describe the system and an evaluation study providing evidence that such an interactive multimodal interface can assist in maintaining situation awareness for operators of autonomous systems, irrespective of cognitive styles.

References

[1]

Julie A Adams . 2007. Unmanned vehicle situation awareness: A path forward Human systems integration symposium. 31--89.

[2]

Marilyn Jager Adams, Yvette J Tenney, and Richard W Pew . 1995. Situation awareness and the cognitive management of complex systems. Human factors, Vol. 37, 1 (1995), 85--104.

[3]

Ann Austin and Jose Abdelnour Nocera . 2015. So, Who Exactly IS The HCI Professional?. In Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems. ACM, 2732906, 1037--1042.

Digital Library

[4]

Vaishak Belle . 2017. Logic meets Probability: Towards Explainable AI Systems for Uncertain Worlds Proceedings of the 26th International Joint Conference on Artificial Intelligence (IJCAI'17). 5116--5120.

Digital Library

[5]

Olesya Blazhenkova and Maria Kozhevnikov . 2009. The new object-spatial-verbal cognitive style model: Theory and measurement. Applied Cognitive Psychology Vol. 23, 5 (2009), 638--663.

[6]

Olessia Blazhenkova, Maria Kozhevnikov, and Michael A. Motes . 2006. Object-spatial imagery: a new self-report imagery questionnaire. Applied Cognitive Psychology Vol. 20, 2 (2006), 239--263.

[7]

Antoine Bordes and Jason Weston . 2016. Learning End-to-End Goal-Oriented Dialog. arXiv preprint (2016). showeprint{arxiv}1605.07683 http://arxiv.org/abs/1605.07683

[8]

John Brooke . 1996. SUS-A quick and dirty usability scale. Usability evaluation in industry. Taylor Francis, London, Chapter 21, 189--194.

[9]

Jing Cheng, Xiao Hu, and P Bryan Heidorn . 2010. New Measures for the Evaluation of Interactive Information Retrieval Systems : Normalized Task Completion Time and Normalized User Effectiveness Proceedings of ASSIST. 1--9.

Digital Library

[10]

Juliet Corbin and Anselm Strauss . 2008. Basics of qualitative research: Techniques and procedures for developing grounded theory. Sage.

[11]

M. L. Cummings . 2004. The Need for Command and Control Instant Message Adaptive Interfaces: Lessons Learned from Tactical Tomahawk Human-in-the-Loop Simulations. Cyberpsychology and behaviour Vol. 7, 6 (2004).

[12]

Francis T. Durso and Scott D. Gronlund . 1999. Situation awareness. Handbook of applied cognition (1999), 283--314.

[13]

Mica R. Endsley . 1988. Design and Evaluation for Situation Awareness Enhancement. Proceedings of the Human Factors Society Annual Meeting, Vol. 32, 2 (1988), 97--101.

[14]

Mica R. Endsley . 1995 a. Measurement of situation awareness in dynamic systems. Human factors, Vol. 37, 1 (1995), 65--84.

[15]

Mica R. Endsley . 1995 b. Toward a theory of situation awareness in dynamic systems. Human factors, Vol. 37, 1 (1995), 32--64.

[16]

Mica R. Endsley and Esin O. Kiris . 1995. The out-of-the-loop performance problem and level of control in automation. Human factors, Vol. 37, 2 (1995), 381--394.

[17]

Yu-Hong Feng, Teck-Hou Teng, and Ah-Hwee Tan . 2009. Modelling situation awareness for Context-aware Decision Support. Expert Systems with Applications Vol. 36, 1 (2009), 455--463.

Digital Library

[18]

Andy Field . 2009. Discovering Statistics Using SPSS (bibinfoedition3rd ed.). Sage, London.

[19]

Kate Forbes-Riley and Diane J. Litman . 2006. Modelling User Satisfaction and Student Learning in a Spoken Dialogue Tutoring System with Generic, Tutoring, and User Affect Parameters Proceedings of NAACL. 264--271.

Digital Library

[20]

Francisco J Chiyah Garcia, David A Robb, Xingkun Liu, Atanas Laskov, Pedro Patron, and Helen Hastie . 2018. Explain Yourself: A Natural Language Interface for Scrutable Autonomous Robots Proceedings of the HRI Workshop on Explainable Robotic Systems, HRI'18.

[21]

Erhan Selcuk Haciomeroglu . 2016. Object-spatial Visualization and Verbal Cognitive Styles, and Their Relation to Cognitive Abilities and Mathematical Performance. Educational Sciences: Theory and Practice Vol. 16, 3 (2016), 987--1003.

[22]

Erhan Selcuk Haciomeroglu and Mark LaVenia . 2017. Object-Spatial Imagery and Verbal Cognitive Styles in High School Students. Perceptual and Motor Skills (2017), 0031512517698555.

[23]

Helen Hastie . 2012. Metrics and evaluation of spoken dialogue systems. Data-Driven Methods for Adaptive Spoken Dialogue Systems. Springer New York, 131--150.

[24]

Helen Hastie, Marie-Aude Aufaure, Panos Alexopoulos, Hugues Bouchard, Catherine Breslin, Heriberto Cuayáhuitl, Nina Dethlefs, Milica Gavsić, James Henderson, Oliver Lemon, et almbox. . 2014. The PARLANCE mobile application for interactive search in English and Mandarin Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL). 260.

[25]

Helen Hastie, Francisco Javier Chiyah Garcia, David A. Robb, Pedro Patron, and Atanas Laskov . 2017 a. MIRIAM: A Multimodal Chat-Based Interface for Autonomous Systems Proceedings of the 19th ACM International Conference on Multimodal Interaction, ICMI'17. ACM, 495--496.

Digital Library

[26]

Helen Hastie, Francisco J. Chiyah Garcia, David A. Robb, Pedro Patron, and Atanas Laskov . 2018 a. MIRIAM: A Multimodal Interface for Explaining the Reasoning Behind Actions of Remote Autonomous Systems. In Proceedings of the ACM 2018 International Conference on Multimodal Interaction (ICMI '18),. ACM, New York, NY, USA.

Digital Library

[27]

Helen Hastie, Xingkun Liu, and Pedro Patron . 2017 b. Trust triggers for multimodal command and control interfaces Proceedings of the 19th ACM International Conference on Multimodal Interaction, ICMI'17. ACM, 261--268.

Digital Library

[28]

Helen Hastie, Katrin Lohan, Mike J. Chantler, David A. Robb, Subramanian Ramamoorthy, Ron Petrick, Sethu Vijayakumar, and David Lane . 2018 b. The ORCA Hub: Explainable Offshore Robotics through Intelligent Interfaces Proceedings of the HRI Workshop on Explainable Robotic Systems, HRI'18.

[29]

Brian Kidwell, Gloria L. Calhoun, Heath A. Ruff, and Raja Parasuraman . 2012. Adaptable and Adaptive Automation for Supervisory Control of Multiple Autonomous Vehicles. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, Vol. 56, 1 (2012), 428--432.

[30]

Maria Kozhevnikov, Olesay Blazhenkova, and Michael Becker . 2010. Trade-off in object versus spatial visualization abilities:Restriction in the development of visual-processing resources. Psychonomic Bulletin and Review Vol. 17, 1 (2010), 29--35.

[31]

David Lane, Keith Brown, Yvan Petillot, Emilio Miguelanez, and Pedro Patron . 2013. An Ontology-Based Approach to Fault Tolerant Mission Execution for Autonomous Platforms. Springer New York, New York, NY, 225--255.

[32]

Oliver Lemon, Anne Bracy, Alexander Gruenstein, and Stanley Peters . 2001. The WITAS multi-modal dialogue system. In Proceedings of the Seventh European Conference on Speech Communication and Technology. 1559--1562.

[33]

Oliver Lemon and Olivier Pietquin . 2012. Data-driven Methods for Adaptive Spoken Dialogue Systems: Computational Learning for Conversational Interfaces. Springer.

Digital Library

[34]

Ivan Maza, Fernando Caballero, R. Molina, N. Pena, and A. Ollero . 2010. Multimodal interface technologies for UAV ground control stations A Comparative Analysis. Journal of Intelligent and Robotic Systems Vol. 57 (2010), 371--391.

Digital Library

[35]

Emily W. Medina, Sunny Fugate, LorRaine Duffy, Dennis Magsombol, Omar Amezcua, Gary Rogers, and Marion G. Ceruti . 2007. Next-Generation Tactical-Situation-Assessment Technology (TSAT): Chat Web Engineering, bibfieldeditorLuciano Baresi, Piero Fraternali, and Geert-Jan Houben (Eds.). Springer Berlin Heidelberg.

[36]

Emilio Miguelanez, Pedro Patron, Keith E Brown, Yvan R Petillot, and David M Lane . 2011. Semantic knowledge-based framework to improve the situation awareness of autonomous underwater vehicles. IEEE Transactions on Knowledge and Data Engineering, Vol. 23, 5 (2011), 759--773.

Digital Library

[37]

Sharon Oviatt and Philip Cohen . 2000. Perceptual user interfaces: multimodal interfaces that process what comes naturally. Commun. ACM, Vol. 43, 3 (2000), 45--53.

Digital Library

[38]

Yvan Petillot, Chris Sotzing, Pedro Patron, David Lane, and Joel Cartright . 2009. Multiple system collaborative planning and sensing for autonomous platforms with shared and distributed situational awareness. Proceedings of the AUVSI's Unmanned Systems Europe, La Spezia, Italy (2009).

[39]

Stephen Rayner and Richard Riding . 1997. Towards a categorisation of cognitive styles and learning styles. Educational Psychology Vol. 17, 1--2 (1997), 5--27.

[40]

David A. Robb, Stefano Padilla, Thomas S. Methven, Britta Kalkreuter, and Mike J. Chantler . 2017. Image-based Emotion Feedback: How Does the Crowd Feel? And Why? Proceedings of the 2017 ACM Conference on Designing Interactive Systems, DIS'17. ACM, 451--463.

Digital Library

[41]

Eugene Sadler-Smith . 2001. The relationship between learning style and cognitive style. Personality and individual differences Vol. 30, 4 (2001), 609--616.

[42]

Iulian Vlad Serban, Ryan Lowe, Laurent Charlin, and Joelle Pineau . 2016. Generative Deep Neural Networks for Dialogue: A Short Review. CoRR Vol. abs/1611.06216 (2016). showeprint{arxiv}1611.06216 http://arxiv.org/abs/1611.06216

[43]

Kip Smith and Peter A Hancock . 1995. Situation awareness is adaptive, externally directed consciousness. Human Factors, Vol. 37, 1 (1995), 137--148.

[44]

Neville A. Stanton, Paul M. Salmon, Guy H. Walker, and Daniel Jenkins . 2009. Genotype and phenotype schemata as models of situation awareness in dynamic command and control teams. International Journal of Industrial Ergonomics, Vol. 39, 3 (2009), 480--489.

[45]

Manfred Stede and David Schlangen . 2004. Information-Seeking Chat: Dialogue Management by Topic Structure Proceedings of the 8th Workshop on the Semantics and Pragmatics of Dialogue. 117---124. https://pdfs.semanticscholar.org/a629/abf9b8a896968b6f9db0baf234f8401ccb90.pdf

[46]

Anselm L Strauss . 1987. Qualitative analysis for social scientists. Cambridge University Press.

[47]

The Institute of Engineering and Technology . 2015. Engineering and Technology Skills and demand in industry 2015 survey. Overview of issues and trends from 2015 survey. Technical Report. https://www.theiet.org/factfiles/education/skills2015-page.cfm Accessed on 6th Aug.2018.

[48]

Oriol Vinyals and Quoc Le . 2015. A neural conversational model. arXiv preprint arXiv:1506.05869 (2015).

[49]

Marilyn A. Walker, Diane J. Litman, Candace A. Kamm, and Alicia Abella . 1997. PARADISE: a framework for evaluating spoken dialogue agents Association for Computational Linguistics. 271--280.

[50]

Marilyn A. Walker, Alex Rudnicky, Rashmi Prasad, John Aberdeen, Elizabeth Owen Bratt, John Garofolo, Helen Hastie, Audrey Le, Bryan Pellom, Alex Potamianos, Rebecca Passonneau, Salim Roukos, Greg S, Stephanie Seneff, and Dave Stallard . 2002. DARPA Communicator: Cross-System Results for the 2001 Evaluation Proceedings of ICSLP 2002. 269--272.

[51]

Christopher D Wickens . 1996. Situation awareness: impact of automation and display technology. Situation awareness: limitations and enhancement in the aviation environment (1996), k2.

Cited By

Roy SSmith TColtin BWilliams TCastellano GRiek LCakmak MLeite I(2023)I Need Your Help... or Do I?Proceedings of the 2023 ACM/IEEE International Conference on Human-Robot Interaction10.1145/3568162.3576954(122-131)Online publication date: 13-Mar-2023
https://dl.acm.org/doi/10.1145/3568162.3576954
Chuang HCheng D(2022)Conversational AI over Military Scenarios Using Intent Detection and Response GenerationApplied Sciences10.3390/app1205249412:5(2494)Online publication date: 27-Feb-2022
https://doi.org/10.3390/app12052494
Linkov VVanžura M(2021)Situation Awareness Measurement in Remotely Controlled CarsFrontiers in Psychology10.3389/fpsyg.2021.59293012Online publication date: 20-Apr-2021
https://doi.org/10.3389/fpsyg.2021.592930
Show More Cited By

Index Terms

Keep Me in the Loop: Increasing Operator Situation Awareness through a Conversational Multimodal Interface
1. Computer systems organization
  1. Embedded and cyber-physical systems
    1. Robotics
      1. Robotic autonomy
2. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction paradigms
      1. Graphical user interfaces
      2. Natural language interfaces

Recommendations

MIRIAM: a multimodal chat-based interface for autonomous systems
ICMI '17: Proceedings of the 19th ACM International Conference on Multimodal Interaction

We present MIRIAM (Multimodal Intelligent inteRactIon for Autonomous systeMs), a multimodal interface to support situation awareness of autonomous vehicles through chat-based interaction. The user is able to chat about the vehicle's plan, objectives, ...
MIRIAM: A Multimodal Interface for Explaining the Reasoning Behind Actions of Remote Autonomous Systems
ICMI '18: Proceedings of the 20th ACM International Conference on Multimodal Interaction

Autonomous systems in remote locations have a high degree of autonomy and there is a need to explain what they are doing and why , in order to increase transparency and maintain trust. This is particularly important in hazardous, high-risk scenarios. ...
Embodiment in conversational interfaces: Rea
CHI '99: Proceedings of the SIGCHI conference on Human Factors in Computing Systems

In this paper, we argue for embodied corrversational characters as the logical extension of the metaphor of human - computer interaction as a conversation. We argue that the only way to fully model the richness of human I&+ to-face communication is to rely ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

ICMI '18: Proceedings of the 20th ACM International Conference on Multimodal Interaction

October 2018

687 pages

ISBN:9781450356923

DOI:10.1145/3242969

General Chairs:
Sidney K. D'Mello
University of Illinois, USA
,
Panayiotis (Panos) Georgiou
University of Southern California, USA
,
Stefan Scherer
University of Southern California, USA
,
Program Chairs:
Emily Mower Provost
University of Michigan, USA
,
Mohammad Soleymani
University of Southern California, USA
,
Marcelo Worsley
Northwestern University, USA

Copyright © 2018 Owner/Author.

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs International 4.0 License.

Sponsors

SIGCHI: Specialist Interest Group in Computer-Human Interaction of the ACM

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 02 October 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Engineering and Physical Sciences Research Council
UK Ministry of Defence
Leverhulme Trust

Conference

ICMI '18

Sponsor:

SIGCHI

ICMI '18: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION

October 16 - 20, 2018

CO, Boulder, USA

Acceptance Rates

ICMI '18 Paper Acceptance Rate 63 of 149 submissions, 42%;

Overall Acceptance Rate 453 of 1,080 submissions, 42%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

11
Total Citations
View Citations
819
Total Downloads

Downloads (Last 12 months)132
Downloads (Last 6 weeks)15

Reflects downloads up to 30 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Roy SSmith TColtin BWilliams TCastellano GRiek LCakmak MLeite I(2023)I Need Your Help... or Do I?Proceedings of the 2023 ACM/IEEE International Conference on Human-Robot Interaction10.1145/3568162.3576954(122-131)Online publication date: 13-Mar-2023
https://dl.acm.org/doi/10.1145/3568162.3576954
Chuang HCheng D(2022)Conversational AI over Military Scenarios Using Intent Detection and Response GenerationApplied Sciences10.3390/app1205249412:5(2494)Online publication date: 27-Feb-2022
https://doi.org/10.3390/app12052494
Linkov VVanžura M(2021)Situation Awareness Measurement in Remotely Controlled CarsFrontiers in Psychology10.3389/fpsyg.2021.59293012Online publication date: 20-Apr-2021
https://doi.org/10.3389/fpsyg.2021.592930
Chiyah Garcia FSmith SLopes JRamamoorthy SHastie HBethel CPaiva ABroadbent EFeil-Seifer DSzafir D(2021)Self-Explainable Robots in Remote EnvironmentsCompanion of the 2021 ACM/IEEE International Conference on Human-Robot Interaction10.1145/3434074.3447275(662-664)Online publication date: 8-Mar-2021
https://dl.acm.org/doi/10.1145/3434074.3447275
Pairet ÈArdón PLiu XLopes JHastie HLohan KKim JTapus ASirkin DJung MKwak S(2019)A digital twin for human-robot interactionProceedings of the 14th ACM/IEEE International Conference on Human-Robot Interaction10.5555/3378680.3378733(372-372)Online publication date: 11-Mar-2019
https://dl.acm.org/doi/10.5555/3378680.3378733
Robb DLopes JPadilla SLaskov AChiyah Garcia FLiu XScharff Willners JValeyrie NLohan KLane DPatron PPetillot YChantler MHastie HHarrison SBardzell SNeustaedter CTatar D(2019)Exploring Interaction with Remote Autonomous Systems using Conversational AgentsProceedings of the 2019 on Designing Interactive Systems Conference10.1145/3322276.3322318(1543-1556)Online publication date: 18-Jun-2019
https://dl.acm.org/doi/10.1145/3322276.3322318
Blumreiter MGreenyer JGarcia FKlös VSchwammberger MSommer CVogelsang AWortmann APretschner AVoss SBurgueño L(2019)Towards self-explainable cyber-physical systemsProceedings of the 22nd International Conference on Model Driven Engineering Languages and Systems10.1109/MODELS-C.2019.00084(543-548)Online publication date: 15-Sep-2019
https://dl.acm.org/doi/10.1109/MODELS-C.2019.00084
Pairet EArdon PLiu XLopes JHastie HLohan K(2019)A Digital Twin for Human-Robot Interaction2019 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI)10.1109/HRI.2019.8673015(372-372)Online publication date: Mar-2019
https://doi.org/10.1109/HRI.2019.8673015
Lopes JLohan KHastie H(2018)Symptoms of cognitive load in interactions with a dialogue systemProceedings of the Workshop on Modeling Cognitive Processes from Multimodal Data10.1145/3279810.3279851(1-5)Online publication date: 16-Oct-2018
https://dl.acm.org/doi/10.1145/3279810.3279851
Hastie HChiyah Garcia FRobb DLaskov APatron PD'Mello SGeorgiou PScherer SProvost ESoleymani MWorsley M(2018)MIRIAMProceedings of the 20th ACM International Conference on Multimodal Interaction10.1145/3242969.3266297(557-558)Online publication date: 2-Oct-2018
https://dl.acm.org/doi/10.1145/3242969.3266297
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents