research-article

Open access

Mixed-initiative Variable Autonomy for Remotely Operated Mobile Robots

Authors:

Rustam StolkinAuthors Info & Claims

ACM Transactions on Human-Robot Interaction (THRI), Volume 10, Issue 4

Article No.: 37, Pages 1 - 34

https://doi.org/10.1145/3472206

Published: 02 September 2021 Publication History

All formats PDF

Abstract

This article presents an Expert-guided Mixed-initiative Control Switcher (EMICS) for remotely operated mobile robots. The EMICS enables switching between different levels of autonomy during task execution initiated by either the human operator and/or the EMICS. The EMICS is evaluated in two disaster-response-inspired experiments, one with a simulated robot and test arena, and one with a real robot in a realistic environment. Analyses from the two experiments provide evidence that: (a) Human-Initiative (HI) systems outperform systems with single modes of operation, such as pure teleoperation, in navigation tasks; (b) in the context of the simulated robot experiment, Mixed-initiative (MI) systems provide improved performance in navigation tasks, improved operator performance in cognitive demanding secondary tasks, and improved operator workload compared to HI. Last, our experiment on a physical robot provides empirical evidence that identify two major challenges for MI control: (a) the design of context-aware MI control systems; and (b) the conflict for control between the robot’s MI control system and the operator. Insights regarding these challenges are discussed and ways to tackle them are proposed.

References

[1]

Julie A. Adams, Pramila Rani, and Nilanjan Sarkar. 2004. Mixed initiative interaction and robotic systems. In Proceedings of the AAAI Workshop on Supervisory Control of Learning and Adaptive Systems. 6–13.

[2]

Michael Baker and Holly A. Yanco. 2004. Autonomy mode suggestions for improving human-robot interaction. In Proceedings of the IEEE International Conference on Systems, Man and Cybernetics (SMC’04), Vol. 3. 2948–2953.

[3]

Robert Goodell Brown. 1963. Smoothing, Forecasting and Prediction of Discrete Time Series. Courier Corporation. 1–468.

[4]

David J. Bruemmer, Douglas A. Few, Ronald L. Boring, Julie L. Marble, Miles C. Walton, and Curtis W. Nielsen. 2005. Shared understanding for collaborative control. IEEE Trans. Syst. Man Cybernet.—Part A: Syst. Hum. 35, 4 (July 2005), 494–504.

[5]

David J. Bruemmer, Julie L. Marble, Donald D. Dudenhoeffer, Matthew O. Anderson, and Mark D. Mckay. 2003. Mixed-initiative control for remote characterization of hazardous environments. In Proceedings of the 36th Annual Hawaii International Conference on System Sciences. 9.

[6]

Jennifer L. Burke, Robin R. Murphy, Michael D. Coovert, and Dawn L. Riddle. 2004. Moonlight in Miami: Field study of human-robot interaction in the context of an urban search and rescue disaster response training exercise. Hum.-Comput. Interact. 19, 1–2 (2004), 85–116.

[7]

Tom Carlson and Yiannis Demiris. 2008. Collaborative control in human–wheelchair interaction reduces the need for dexterity in precise manoeuvres. In Proceedings of Robotic Helpers: User Interaction, Interfaces and Companions in Assistive and Therapy Robotics, a Workshop at ACM/IEEE HRI. 59–66.

[8]

Jennifer L. Casper and Robin R. Murphy. 2003. Human-robot interactions during the robot-assisted urban search and rescue response at the world trade center. IEEE Trans. Syst. Man Cybernet.—Part B: Cybernet. 33, 3 (2003), 367–385.

[9]

Caroline P. Carvalho Chanel, Raphaëlle N. Roy, Frédéric Dehais, and Nicolas Drougard. 2020. Towards mixed-initiative human–robot interaction: Assessment of discriminative physiological and behavioral features for performance prediction. Sensors 20, 1 (2020), 296. https://doi.org/10.3390/s20010296

[10]

Manolis Chiou, Goda Bieksaite, Nick Hawes, and Rustam Stolkin. 2016. Human-initiative variable autonomy: An experimental analysis of the interactions between a human operator and a remotely operated mobile robot which also possesses autonomous capabilities. In Proceedings of the AAAI Fall Symposium Series: Shared Autonomy in Research and Practice. 304–310.

[11]

Manolis Chiou, Goda Bieksaite, Rustam Stolkin, Nick Hawes, Kimron L. Shapiro, and Timothy S. Harrison. 2016. Experimental analysis of a variable autonomy framework for controlling a remotely operating mobile robot. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS’16). 3581–3588.

[12]

Manolis Chiou, Nick Hawes, Rustam Stolkin, Kimron L. Shapiro, Jess R. Kerlin, and Andrew Clouter. 2015. Towards the principled study of variable autonomy in mobile robots. In Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics (SMC’15). 1053–1059.

Digital Library

[13]

Manolis Chiou, Mohammed Talha, and Rustam Stolkin. 2019. Learning effects in variable autonomy human-robot systems: How much training is enough? In Proceedings of the IEEE International Conference on Systems, Man and Cybernetics (SMC’19). 720–727. https://doi.org/10.1109/SMC.2019.8914558

Digital Library

[14]

Lorin D. Dole, David M. Sirkin, Robin R. Murphy, and Clifford I. Nass. 2015. Robots need humans in the loop to improve the hopefulness of disaster survivors. In Proceedings of the IEEE International Workshop on Robot and Human Interactive Communication (RO-MAN’15). 707–714.

[15]

Anca D. Dragan and Siddhartha S. Srinivasa. 2013. A policy-blending formalism for shared control. Int. J. Robot. Res. 32, 7 (2013), 790–805. https://doi.org/10.1177/0278364913490324

Digital Library

[16]

Douglas A. Few, David J. Bruemmer, and Miles Walton. 2006. Improved human-robot teaming through facilitated initiative. In Proceedings of the IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN’06). 171–176.

[17]

Alberto Finzi and Andrea Orlandini. 2005. Human-robot interaction through mixed-initiative planning for rescue and search rovers. In Proceedings of the Congress of the Italian Association for Artificial Intelligence. 483–494.

Digital Library

[18]

Giorgio Ganis and Rogier Kievit. 2015. A new set of three-dimensional shapes for investigating mental rotation processes: Validation data and stimulus set. J. Open Psychol. Data 3, 1 (2015).

[19]

Thibault Gateau, Caroline P. Carvalho Chanel, Mai-huy Le, and Frederic Dehais. 2016. Considering human’s non-deterministic behavior and his availability state when designing a collaborative human-robots system. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS’16). 4391–4397. https://doi.org/10.1109/IROS.2016.7759646

Digital Library

[20]

Matthew Gombolay, Anna Bair, Cindy Huang, and Julie Shah. 2017. Computational design of mixed-initiative human–robot teaming that considers human factors: Situational awareness, workload, and workflow preferences. Int. J. Robot. Res. 36, 5–7 (2017), 598–617.

Digital Library

[21]

Matthew C. Gombolay, Reymundo A. Gutierrez, Shanelle G. Clarke, Giancarlo F. Sturla, and Julie A. Shah. 2015. Decision-making authority, team efficiency and human worker satisfaction in mixed human–robot teams. Auton. Robots 39, 3 (2015), 293–312. https://doi.org/10.1007/s10514-015-9457-9

Digital Library

[22]

Michael A. Goodrich, Dan R. Olsen Jr., Jacob W. Crandall, and Thomas J. Palmer. 2001. Experiments in adjustable autonomy. In Proceedings of the IJCAI Workshop on Autonomy, Delegation, and Control: Interacting with Autonomous Agents. 1624–1629.

[23]

Benjamin Hardin and Michael A. Goodrich. 2009. On using mixed-initiative control: A perspective for managing large-scale robotic teams. In Proceedings of the 4th ACM/IEEE International Conference on Human–Robot Interaction (HRI’09). 165–172.

[24]

A. Hong, O. Igharoro, Y. Liu, F. Niroui, G. Nejat, and B. Benhabib. 2019. Investigating human-robot teams for learning-based semi-autonomous control in urban search and rescue environments. J. Intell. Robot. Syst. 94, 3–4 (2019), 669–686. https://doi.org/10.1007/s10846-018-0899-0

Digital Library

[25]

A. Jacoff, E. Messina, B. A. Weiss, S. Tadokoro, and Y. Nakagawa. 2003. Test arenas and performance metrics for urban search and rescue robots. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS’03), Vol. 3. 3396–3403.

[26]

Siddarth Jain and Brenna Argall. 2019. Probabilistic human intent recognition for shared autonomy in assistive robotics. ACM Trans. Hum.-Robot Interact. 9, 1 (2019), 1–23. https://doi.org/10.1145/3359614

Digital Library

[27]

Shu Jiang and Ronald C. Arkin. 2015. Mixed-initiative human-robot interaction: Definition, taxonomy, and Survey. In Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics (SMC’15). 954–961.

[28]

Aaron W. Johnson, Kevin R. Duda, Thomas B. Sheridan, and Charles M. Oman. 2017. A closed-loop model of operator visual attention, situation awareness, and performance across automation mode transitions. Hum. Factors 59, 2 (2017), 229–241.

[29]

Yugang Liu and Goldie Nejat. 2016. Multirobot cooperative learning for semiautonomous control in urban search and rescue applications. J. Field Robot. 33, 4 (2016), 512–536. https://doi.org/10.1002/rob.21597

Digital Library

[30]

E. H. Mamdani and S. Assilian. 1975. An experiment in linguistic synthesis with a fuzzy logic controller. Int. J. Man-Mach. Studies 7, 1 (1975), 1–13.

[31]

Vikram Manikonda, Priya Ranjan, Zachary Kulis, and Calhoun Drive. 2007. A mixed initiative controller and testbed for human–robot teams in tactical operations. In Proceedings of the AAAI Fall Symposium. 92–99.

[32]

Julie L. Marble, David J. Bruemmer, Douglas A. Few, and Donald D. Dudenhoeffer. 2004. Evaluation of supervisory vs. peer-peer interaction with human-robot teams. In Proceedings of the 37th Hawaii International Conference on System Sciences. 1–9.

[33]

Eitan Marder-Eppstein, Eric Berger, Tully Foote, Brian Gerkey, and Kurt Konolige. 2010. The office marathon: Robust navigation in an indoor office environment. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA’10). 300–307.

[34]

Robin R. Murphy. 2004. Human-robot interaction in rescue robotics. IEEE Trans. Syst. Man Cybernet. Part C (Appl. Rev.) 34, 2 (May 2004), 138–153.

Digital Library

[35]

Robin R. Murphy and Jennifer L. Burke. 2005. Up from the rubble: Lessons learned about HRI from Search and Rescue. In Proceedings of the Human Factors and Ergonomics Society 49th Annual Meeting. 437–441.

[36]

Selma Musić and Sandra Hirche. 2017. Control sharing in human-robot team interaction. Annu. Rev. Control 44 (2017), 342–354. https://doi.org/10.1016/j.arcontrol.2017.09.017

[37]

Keiji Nagatani, Seiga Kiribayashi, Yoshito Okada, Kazuki Otake, Kazuya Yoshida, Takeshi Nishimura, Tomoaki Yoshida, Eiji Koyanagi, Satoshi Tadokoro, Mineo Fukushima, and Shinji Kawatsuma. 2013. Emergency response to the nuclear accident at the Fukushima Daiichi nuclear power plants using mobile rescue robots. J. Field Robot. 30, 1 (2013), 44–63.

Digital Library

[38]

Farrukh Nagi, L. Perumal, and J. Nagi. 2009. A new integrated fuzzy bang-bang relay control system. Mechatronics 19, 5 (2009), 748–760.

[39]

Curtis W. Nielsen, Douglas A. Few, and Devin S. Athey. 2008. Using mixed-initiative human-robot interaction to bound performance in a search task. In Proceedings of the IEEE International Conference on Intelligent Sensors, Sensor Networks and Information Processing. 195–200.

[40]

Adam Norton, Willard Ober, Lisa Baraniecki, Eric McCann, Jean Scholtz, David Shane, Anna Skinner, Robert Watson, and Holly Yanco. 2017. Analysis of human–robot interaction at the DARPA Robotics Challenge Finals. Int. J. Robot. Res. 36, 5–7 (2017), 483–513. https://doi.org/10.1177/0278364916688254

Digital Library

[41]

Thomas V. Perneger. 1998. What’s wrong with Bonferroni adjustments.BMJ Clin. Res. Ed. 316, 7139 (1998), 1236–8.

[42]

Juan Rada-Vilela. 2013. fuzzylite: A fuzzy logic control library in C++. In Proceedings of the Open Source Developers Conference.

[43]

Marc Rigter, Bruno Lacerda, and Nick Hawes. 2020. A framework for learning from demonstration with minimal human effort. IEEE Robot. Autom. Lett. 5, 2 (2020), 2023–2030. https://doi.org/10.1109/LRA.2020.2970619

[44]

Simon Rothfuss, Robert Schmidt, Michael Flad, and Soren Hohmann. 2019. A concept for human-machine negotiation in advanced driving assistance systems. In Proceedings of the IEEE International Conference on Systems, Man and Cybernetics (SMC’19). 3116–3123. https://doi.org/10.1109/SMC.2019.8914282

Digital Library

[45]

Kenneth J. Rothman. 1990. No adjustments are needed for multiple comparisons. Epidemiology 1, 1 (1990), 43–46.

[46]

David Sharek. 2011. A useable, online NASA-TLX tool. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting. 1375–1379.

[47]

Jian Shen, Javier Ibanez-Guzman, Teck Chew Ng, and Boon Seng Chew. 2004. A collaborative-shared control system with safe obstacle avoidance capability. In Proceedings of the IEEE Conference on Robotics, Automation and Mechatronics, Vol. 1. 119–123.

[48]

Thomas B. Sheridan and William L. Verplank. 1978. Human and computer control of undersea teleoperators. Technical report, MIT Man-Machine Systems Laboratory.

[49]

David L. Streiner and Geoffrey R. Norman. 2011. Correction for multiple testing: Is there a resolution?CHEST J. 140, 1 (2011), 16.

[50]

Holly A. Yanco, Adam Norton, Willard Ober, David Shane, Anna Skinner, and Jack Vice. 2015. Analysis of human-robot interaction at the DARPA robotics challenge trials. J. Field Robot. 32, 3 (2015), 420–444.

Digital Library

Cited By

Natarajan MXue Cvan Waveren SFeigh KGombolay MDastani MSichman JAlechina NDignum V(2024)Mixed-Initiative Human-Robot Teaming under Suboptimality with Online Bayesian AdaptationProceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems10.5555/3635637.3663005(1454-1462)Online publication date: 6-May-2024
https://dl.acm.org/doi/10.5555/3635637.3663005
Verhagen RNeerincx MTielman M(2024)Meaningful human control and variable autonomy in human-robot teams for firefightingFrontiers in Robotics and AI10.3389/frobt.2024.132398011Online publication date: 1-Feb-2024
https://doi.org/10.3389/frobt.2024.1323980
Conlon NAhmed NSzafir D(2024)Event-triggered robot self-assessment to aid in autonomy adjustmentFrontiers in Robotics and AI10.3389/frobt.2023.129453310Online publication date: 4-Jan-2024
https://doi.org/10.3389/frobt.2023.1294533
Show More Cited By

Index Terms

Mixed-initiative Variable Autonomy for Remotely Operated Mobile Robots

Recommendations

Autonomy, Embodiment, and Obedience to Robots
HRI'15 Extended Abstracts: Proceedings of the Tenth Annual ACM/IEEE International Conference on Human-Robot Interaction Extended Abstracts

We conducted an HRI obedience experiment comparing an autonomous robotic authority to: (i) a remote-controlled robot, and (ii) robots of variant embodiments during a deterrent task. The results suggest that half of people will continue to perform a ...
A shared control method for obstacle avoidance with mobile robots and its interaction with communication delay

Teleoperation allows human operators to safely extend themselves to remote environments that are typically difficult or dangerous to access. The remote environments are often unstructured i.e. not having clear roads or paths to follow and only ...
Design and Simulation of Autonomous Mobile Robots Obstacle Avoidance System
LNCS on Transactions on Edutainment XIII - Volume 10092

Autonomous mobile robot is a completely unknown or partially unknown, of its complex environment, which requires the ability to avoid obstacles in order to move safely and avoid the collisions while navigating. To solve this problem, it opted to propose ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Human-Robot Interaction

ACM Transactions on Human-Robot Interaction Volume 10, Issue 4

December 2021

282 pages

EISSN:2573-9522

DOI:10.1145/3476005

Editors:
Odest Chadwicke Jenkins
University of Michigan, USA
,
Selma Sabanovic
Indiana University, USA

Issue’s Table of Contents

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 02 September 2021

Accepted: 01 May 2021

Revised: 01 January 2021

Received: 01 February 2020

Published in THRI Volume 10, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

Defence Science and Technology Laboratory (Dstl)
Royal Society Industry Fellowship
UK’s Engineering and Physical Sciences Research Council (EPSRC)

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

22
Total Citations
View Citations
1,364
Total Downloads

Downloads (Last 12 months)458
Downloads (Last 6 weeks)109

Reflects downloads up to 09 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Natarajan MXue Cvan Waveren SFeigh KGombolay MDastani MSichman JAlechina NDignum V(2024)Mixed-Initiative Human-Robot Teaming under Suboptimality with Online Bayesian AdaptationProceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems10.5555/3635637.3663005(1454-1462)Online publication date: 6-May-2024
https://dl.acm.org/doi/10.5555/3635637.3663005
Verhagen RNeerincx MTielman M(2024)Meaningful human control and variable autonomy in human-robot teams for firefightingFrontiers in Robotics and AI10.3389/frobt.2024.132398011Online publication date: 1-Feb-2024
https://doi.org/10.3389/frobt.2024.1323980
Conlon NAhmed NSzafir D(2024)Event-triggered robot self-assessment to aid in autonomy adjustmentFrontiers in Robotics and AI10.3389/frobt.2023.129453310Online publication date: 4-Jan-2024
https://doi.org/10.3389/frobt.2023.1294533
Methnani LChiou MDignum VTheodorou A(2024)Who’s in Charge Here? A Survey on Trustworthy AI in Variable Autonomy Robotic SystemsACM Computing Surveys10.1145/364509056:7(1-32)Online publication date: 9-Apr-2024
https://dl.acm.org/doi/10.1145/3645090
Reinmund TSalvini PKunze LJirotka MWinfield A(2024)Variable Autonomy through Responsible Robotics: Design Guidelines and Research AgendaACM Transactions on Human-Robot Interaction10.1145/363643213:1(1-36)Online publication date: 30-Jan-2024
https://dl.acm.org/doi/10.1145/3636432
Li YCui RYan WZhang SYang C(2024)Reconciling Conflicting Intents: Bidirectional Trust-Based Variable Autonomy for Mobile RobotsIEEE Robotics and Automation Letters10.1109/LRA.2024.33961009:6(5615-5622)Online publication date: Jun-2024
https://doi.org/10.1109/LRA.2024.3396100
Schlafly MPrabhakar APopovic KSchlafly GKim CMurphey T(2024)Collaborative robots can augment human cognition in regret-sensitive tasksPNAS Nexus10.1093/pnasnexus/pgae0163:2Online publication date: 17-Jan-2024
https://doi.org/10.1093/pnasnexus/pgae016
Chiwaridzo OMasengu R(2024)Rebuilding sustainable green tourism supply chain through technology adoption and social media branding in Zimbabwe post-COVID-19.Environment, Development and Sustainability10.1007/s10668-024-04718-yOnline publication date: 12-Mar-2024
https://doi.org/10.1007/s10668-024-04718-y
Guillet VGrand CLesire CPicard G(2024)Bid Intercession to Unlock Human Control in Decentralized Consensus-Based Multi-robot Task Allocation AlgorithmsAgents and Robots for reliable Engineered Autonomy10.1007/978-3-031-73180-8_7(99-114)Online publication date: 13-Oct-2024
https://doi.org/10.1007/978-3-031-73180-8_7
Ramesh ABraun CRuan TRothfuß SHohmann SStolkin RChiou M(2023)Experimental Evaluation of Model Predictive Mixed-Initiative Variable Autonomy Systems Applied to Human-Robot Teams2023 IEEE International Conference on Systems, Man, and Cybernetics (SMC)10.1109/SMC53992.2023.10394184(5291-5298)Online publication date: 1-Oct-2023
https://doi.org/10.1109/SMC53992.2023.10394184
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Media

Figures

Other

Tables

View Issue’s Table of Contents