short-paper

Towards Human-in-the-Loop Autonomous Multi-Robot Operations

Authors:

Marcel Kaufmann,

Katherine Sheridan,

Giovanni BeltrameAuthors Info & Claims

ICMI '21 Companion: Companion Publication of the 2021 International Conference on Multimodal Interaction

Pages 341 - 343

https://doi.org/10.1145/3461615.3486573

Published: 17 December 2021 Publication History

Abstract

Rapid advances in artificial intelligence are driving applications of robotics and automation in transport and logistics, providing new solutions to highway systems, passenger transport, last-mile delivery, and automated warehouses. Because the environment is dynamic and not entirely knowable, human supervision will be needed for the foreseeable future to solve unexpected problems. Research in other domains suggests that human-robot teams may even offer the optimal solution, as the relative strengths of human and artificial intelligence can be combined. However, realizing human-robot complementarity is challenging because sophisticated AI techniques lead to increasingly opaque robot control programs, and operators’ cognitive capacities can be exceeded as the sizes of autonomous fleets grow. We present our vision and initial results for a framework for human supervisory control and human robot teamwork for automated multi-vehicle systems.

References

[1]

Lobna Chérif, Valerie Wood, Alexandre Marois, Katherine Labonté, and François Vachon. 2018. Multitasking in the military: Cognitive consequences and potential solutions. Applied cognitive psychology 32, 4 (2018), 429–439.

[2]

Emily BJ Coffey and Sibylle C Herholz. 2013. Task decomposition: a framework for comparing diverse training models in human brain plasticity studies. Frontiers in human neuroscience 7 (2013), 640.

[3]

Benjamin Cowley, Marco Filetti, Kristian Lukander, Jari Torniainen, Andreas Henelius, Lauri Ahonen, Oswald Barral, Ilkka Kosunen, Teppo Valtonen, Minna Huotilainen, 2016. The psychophysiology primer: a guide to methods and a broad review with a focus on human–computer interaction. Foundations and Trends® in Human–Computer Interaction 9, 3-4(2016), 151–308.

[4]

Mary L Cummings, Andrew Clare, and Christin Hart. 2010. The role of human-automation consensus in multiple unmanned vehicle scheduling. Human Factors 52, 1 (2010), 17–27.

[5]

John J. Enright and Peter R. Wurman. 2011. Optimization and Coordinated Autonomy in Mobile Fulfillment Systems. In Workshops at the Twenty-Fifth AAAI Conference on Artificial Intelligence. https://www.aaai.org/ocs/index.php/WS/AAAIW11/paper/view/3917

[6]

Fei Gao, Mary L Cummings, and Erin Treacy Solovey. 2014. Modeling teamwork in supervisory control of multiple robots. Human-Machine Systems, IEEE Transactions on 44, 4 (2014), 441–453.

[7]

Pasquale Grippa, Doris A. Behrens, Friederike Wall, and Christian Bettstetter. 2019. Drone delivery systems: job assignment and dimensioning. Autonomous Robots 43, 2 (Feb. 2019), 261–274. https://doi.org/10.1007/s10514-018-9768-8

Digital Library

[8]

Nicholas Hyldmar, Yijun He, and Amanda Prorok. 2019. A Fleet of Miniature Cars for Experiments in Cooperative Driving. In IEEE International Conference on Robotics and Automation (ICRA).

Digital Library

[9]

Iring Koch, Edita Poljac, Hermann Müller, and Andrea Kiesel. 2018. Cognitive structure, flexibility, and plasticity in human multitasking—An integrative review of dual-task and task-switching research.Psychological bulletin 144, 6 (2018), 557.

[10]

Andreas Kolling, Phillip Walker, Nilanjan Chakraborty, Katia Sycara, and Michael Lewis. 2016. Human Interaction with Robot Swarms: A Survey. IEEE Transactions on Human-Machine Systems 46, 1 (2016), 9–26. https://doi.org/10.1109/THMS.2015.2480801

[11]

Ryan McKendrick, Tyler Shaw, Ewart de Visser, Haneen Saqer, Brian Kidwell, and Raja Parasuraman. 2013. Team performance in networked supervisory control of unmanned air vehicles effects of automation, working memory, and communication content. Human Factors: The Journal of the Human Factors and Ergonomics Society (2013), 0018720813496269.

[12]

Nicolette Mogilever, Lucrezia Zuccarelli, Ford Burles, Giuseppe Iaria, Giacomo Strapazzon, Loredana Bessone, and Emily BJ Coffey. 2018. Expedition cognition: a review and prospective of subterranean neuroscience with spaceflight applications. Frontiers in human neuroscience 12 (2018), 407.

[13]

Raja Parasuraman, Scott Galster, Peter Squire, Hiroshi Furukawa, and Christopher Miller. 2005. A flexible delegation-type interface enhances system performance in human supervision of multiple robots: Empirical studies with RoboFlag. Systems, Man and Cybernetics, Part A: Systems and Humans, IEEE Transactions on 35, 4(2005), 481–493.

Digital Library

[14]

Marco Pavone. 2015. Autonomous Mobility-on-Demand Systems for Future Urban Mobility. In Autonomes Fahren: Technische, rechtliche und gesellschaftliche Aspekte, Markus Maurer, J. Christian Gerdes, Barbara Lenz, and Hermann Winner (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 399–416. https://doi.org/10.1007/978-3-662-45854-9_19

[15]

Steven E. Shladover. 2018. Connected and automated vehicle systems: Introduction and overview. Journal of Intelligent Transportation Systems 22, 3 (May 2018), 190–200. https://doi.org/10.1080/15472450.2017.1336053

[16]

L Melissa Skaugset, Susan Farrell, Michele Carney, Margaret Wolff, Sally A Santen, Marcia Perry, and Stephen John Cico. 2016. Can you multitask? Evidence and limitations of task switching and multitasking in emergency medicine. Annals of emergency medicine 68, 2 (2016), 189–195.

[17]

Nicolas Valenchon, Yann Bouteiller, Hugo R. Jourde, Emily B. J. Coffey, and Giovanni Beltrame. 2021. The Portiloop: a deep learning-based open science tool for closed-loop brain stimulation. arXiv:arXiv:2107.13473

[18]

Jijun Wang and Michael Lewis. 2007. Human control for cooperating robot teams. In Human-Robot Interaction (HRI), 2007 2nd ACM/IEEE International Conference on. IEEE, 9–16.

Digital Library

[19]

Xiaotian Wang and Yue Wang. 2017. Co-design of Control and Scheduling for Human–Swarm Collaboration Systems Based on Mutual Trust. In Trends in Control and Decision-Making for Human–Robot Collaboration Systems. Springer, 387–413.

[20]

Guang-Zhong Yang, Jim Bellingham, Pierre E Dupont, Peer Fischer, Luciano Floridi, Robert Full, Neil Jacobstein, Vijay Kumar, Marcia McNutt, Robert Merrifield, 2018. The grand challenges of Science Robotics. Science Robotics 3, 14 (2018), eaar7650.

[21]

Haihong Zhang, Yongwei Zhu, Jayachandran Maniyeri, and Cuntai Guan. 2014. Detection of variations in cognitive workload using multi-modality physiological sensors and a large margin unbiased regression machine. In 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE, 2985–2988.

Cited By

Abioye ANaiseh MHunt WClark JRamchurn SSoorati M(2023)The Effect of Data Visualisation Quality and Task Density on Human-Swarm Interaction2023 32nd IEEE International Conference on Robot and Human Interactive Communication (RO-MAN)10.1109/RO-MAN57019.2023.10309454(1494-1501)Online publication date: 28-Aug-2023
https://doi.org/10.1109/RO-MAN57019.2023.10309454
Sandula AKhokhar AGhose DBiswas P(2023)Demand-Aware Multi-Robot Task Scheduling with Mixed Reality Simulation2023 32nd IEEE International Conference on Robot and Human Interactive Communication (RO-MAN)10.1109/RO-MAN57019.2023.10309373(1606-1611)Online publication date: 28-Aug-2023
https://doi.org/10.1109/RO-MAN57019.2023.10309373

Index Terms

Towards Human-in-the-Loop Autonomous Multi-Robot Operations
1. Computer systems organization
  1. Embedded and cyber-physical systems
    1. Robotics
      1. Robotic autonomy
2. Computing methodologies
  1. Artificial intelligence
    1. Control methods
      1. Robotic planning
    2. Planning and scheduling
      1. Robotic planning

Index terms have been assigned to the content through auto-classification.

Recommendations

Towards Autonomous Mobile Robot Navigation
Human-Centered Robot Navigation—Towards a Harmoniously Human–Robot Coexisting Environment

This paper proposes a navigation algorithm that considers the states of humans and robots in order to achieve harmonious coexistence between them. A robot navigation in the presence of humans and other robots is rarely considered in the field of ...
A mobile robot with autonomous climbing and descending of stairs

Mobile robots are used to operate in urban environments, for surveillance, reconnaissance, and inspection, as well as for military operations and in hazardous environments. Some are intended for exploration of only natural terrains, but others also for ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICMI '21 Companion: Companion Publication of the 2021 International Conference on Multimodal Interaction

October 2021

418 pages

ISBN:9781450384711

DOI:10.1145/3461615

Editors:
Zakia Hammal
Carnegie Mellon University
,
Carlos Busso
University of Texas at Dallas
,
Catherine Pelachaud
CNRS - ISIR, Sorbonne University
,
Sharon Oviatt
Monash University
,
Albert Ali Salah
Utrecht University and Boğaziçi University
,
Guoying Zhao
University of Oulu

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 December 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Short-paper
Research
Refereed limited

Funding Sources

New Frontiers Research Fund

Conference

ICMI '21

Sponsor:

SIGCHI

ICMI '21: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION

October 18 - 22, 2021

QC, Montreal, Canada

Acceptance Rates

Overall Acceptance Rate 453 of 1,080 submissions, 42%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
146
Total Downloads

Downloads (Last 12 months)27
Downloads (Last 6 weeks)1

Reflects downloads up to 31 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Abioye ANaiseh MHunt WClark JRamchurn SSoorati M(2023)The Effect of Data Visualisation Quality and Task Density on Human-Swarm Interaction2023 32nd IEEE International Conference on Robot and Human Interactive Communication (RO-MAN)10.1109/RO-MAN57019.2023.10309454(1494-1501)Online publication date: 28-Aug-2023
https://doi.org/10.1109/RO-MAN57019.2023.10309454
Sandula AKhokhar AGhose DBiswas P(2023)Demand-Aware Multi-Robot Task Scheduling with Mixed Reality Simulation2023 32nd IEEE International Conference on Robot and Human Interactive Communication (RO-MAN)10.1109/RO-MAN57019.2023.10309373(1606-1611)Online publication date: 28-Aug-2023
https://doi.org/10.1109/RO-MAN57019.2023.10309373

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View Table of Conten