research-article

Establishing Appropriate Trust via Critical States

Authors:

Sandy H. Huang,

Anca D. DraganAuthors Info & Claims

2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

Pages 3929 - 3936

https://doi.org/10.1109/IROS.2018.8593649

Published: 01 October 2018 Publication History

Abstract

In order to effectively interact with or supervise a robot, humans need to have an accurate mental model of its capabilities and how it acts. Learned neural network policies make that particularly challenging. We propose an approach for helping end-users build a mental model of such policies. Our key observation is that for most tasks, the essence of the policy is captured in a few critical states: states in which it is very important to take a certain action. Our user studies show that if the robot shows a human what its understanding of the task's critical states is, then the human can make a more informed decision about whether to deploy the policy, and if she does deploy it, when she needs to take control from it at execution time.

References

[1]

M. J. Gielniak and A. L. Thomaz, “Generating anticipation in robot motion,” in Proceedings of the Twentieth IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), 2011, pp. 449–454.

[2]

A. D. Dragan, K. C. T. Lee, and S. S. Srinivasa, “Legibility and predictability of robot motion,” in Proceedings of the Eighth ACM/IEEE International Conference on Human-Robot Interaction (HRI). ACM, 2013, pp. 301–308.

[3]

D. Szafir, B. Mutlu, and T. Fong, “Communication of intent in assistive free flyers,” in Proceedings of the Ninth ACM/IEEE International Conference on Human-Robot Interaction (HRI). ACM, 2014, pp. 358–365.

[4]

S. H. Huang, D. Held, P. Abbeel, and A. D. Dragan, “Enabling robots to communicate their objectives,” in Proceedings of Robotics: Science and Systems (RSS), 2017.

[5]

S. Nikolaidis, S. Nath, A. D. Procaccia, and S. Srinivasa, “Game-theoretic modeling of human adaptation in human-robot collaboration,” in Proceedings of the Twelfth ACM/IEEE International Conference on Human-Robot Interaction (HRI). ACM, 2017, pp. 323–331.

[6]

M. Kwon, S. H. Huang, and A. D. Dragan, “Expressing robot incapability,” in Proceedings of the Thirteenth ACM/IEEE International Conference on Human Robot Interaction (HRI). ACM, 2018.

[7]

N. Wang, D. V. Pynadath, and S. G. Hill, “Trust calibration within a human-robot team: Comparing automatically generated explanations,” in Proceedings of the Eleventh ACM/IEEE International Conference on Human Robot Interaction (HRI). ACM, 2016, pp. 109–116.

[8]

M. T. Dzindolet, S. A. Peterson, R. A. Pomranky, L. G. Pierce, and H. P. Beck, “The role of trust in automation reliance,” International Journal of Human-Computer Studies, vol. 58, no. 6, pp. 697–718, June 2003.

Digital Library

[9]

J. D. Lee and K. A. See, “Trust in automation: Designing for appropriate reliance,” Human Factors, vol. 46, no. 1, pp. 50–80, 2004.

[10]

S. Ososky, D. Schuster, E. Phillips, and F. Jentsch, “Building appropriate trust in human-robot teams,” in Proceedings of the Twenty-Seventh AAAI Conference on Artificial Intelligence, 2013.

[11]

A. Freedy, E. DeVisser, G. Weltman, and N. Coeyman, “Measure-ment of trust in human-robot collaboration,” in 2007 International Symposium on Collaborative Technologies and Systems, May 2007, pp. 106–114.

[12]

E. Cha, A. D. Dragan, and S. S. Srinivasa, “Perceived robot capability,” in Proceedings of the Twenty-Fourth IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), 2015, pp. 541–548.

[13]

B. M. Muir, “Trust between humans and machines, and the design of decision aids,” International Journal of Man-Machine Studies, vol. 27, no. 5, pp. 527–539, 1987.

Digital Library

[14]

D. H. McKnight and N. L. Chervany, “What trust means in e-commerce customer relationships: An interdisciplinary conceptual typology,” International Journal of Electronic Commerce, vol. 6, no. 2, pp. 35–59, 2001.

Digital Library

[15]

M. Lewis, K. Sycara, and P. Walker, “The role of trust in human-robot interaction,” in Foundations of Trusted Autonomy, H. A. Abbass, J. Scholz, and D. J. Reid, Eds. Springer International Publishing, 2018, pp. 135–159.

[16]

S. Levine, C. Finn, T. Darrell, and P. Abbeel, “End-to-end training of deep visuomotor policies,” Journal of Machine Learning Research, vol. 17, no. 39, pp. 1–40, 2016.

Digital Library

[17]

M. Bojarski, D. D. Testa, D. Dworakowski, B. Firner, B. Flepp, P. Goyal, L. D. Jackel, M. Monfort, U. Muller, J. Zhang, X. Zhang, J. Zhao, and K. Zieba, “End to end learning for self-driving cars,” arXiv preprint arXiv:, 2016.

[18]

A. Dragan and S. Srinivasa, “Familiarization to robot motion,” in Proceedings of the Ninth ACM/IEEE International Conference on Human-Robot Interaction (HRI). ACM, 2014.

[19]

C. L. Baker, R. Saxe, and J. B. Tenenbaum, “Action understanding as inverse planning,” Cognition, vol. 113, no. 3, p. 329–349, 2009.

[20]

J. Jara-Ettinger, H. Gwen, L. E. Schulz, and J. B. Tenenbaum, “The naïve utility calculus: Computational principles underlying commonsense psychology,” Trends in Cognitive Sciences, vol. 20, no. 8, p. 589–604, 2016.

[21]

B. W. Israelsen and N. R. Ahmed, ‘’…I can assure you…that its going to be all right. A definition, case for, and survey of algorithmic assurances in human-autonomy trust relationships,” arXiv preprint arXiv:, 2017.

[22]

B. D. Ziebart, A. Maas, J. A. Bagnell, and A. Dey, “Maximum entropy inverse reinforcement learning,” in Proceedings of the Twenty-Second AAAI Conference on Artificial Intelligence, 2008.

[23]

T. Haarnoja, H. Tang, P. Abbeel, and S. Levine, “Reinforcement learning with deep energy-based policies,” in International Conference on Machine Learning, 2017.

[24]

T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.” in Neural Information Processing Systems (NIPS) Deep Reinforcement Learning Symposium, 2017.

[25]

R. S. Sutton and A. G. Barto, Introduction to Reinforcement Learning, 1st ed. Cambridge, MA, USA: MIT Press, 1998.

Digital Library

[26]

S. Taheri and E. H. Law, “Investigation of a combined slip control braking and closed loop four wheel steering system for an automobile during combined hard braking and severe steering,” in Proceedings of the American Control Conference, 1990.

Cited By

Yuan ZGuo WJia JLi BSong DSalakhutdinov RKolter ZHeller KWeller AOliver NScarlett JBerkenkamp F(2024)SHINEProceedings of the 41st International Conference on Machine Learning10.5555/3692070.3694458(57887-57904)Online publication date: 21-Jul-2024
https://dl.acm.org/doi/10.5555/3692070.3694458
Mehrotra SDegachi CVereschak OJonker CTielman M(2024)A Systematic Review on Fostering Appropriate Trust in Human-AI Interaction: Trends, Opportunities and ChallengesACM Journal on Responsible Computing10.1145/36964491:4(1-45)Online publication date: 21-Sep-2024
https://dl.acm.org/doi/10.1145/3696449
Milani STopin NVeloso MFang F(2024)Explainable Reinforcement Learning: A Survey and Comparative ReviewACM Computing Surveys10.1145/361686456:7(1-36)Online publication date: 9-Apr-2024
https://dl.acm.org/doi/10.1145/3616864
Show More Cited By

Index Terms

Establishing Appropriate Trust via Critical States

Index terms have been assigned to the content through auto-classification.

Recommendations

Critical infrastructure dependencies

The proper functioning of critical infrastructures is crucial to societal well-being. However, critical infrastructures are not isolated, but instead are tightly coupled, creating a complex system of interconnected infrastructures. Dependencies between ...
Research on Critical Infrastructures and Critical Information Infrastructures
BLISS '09: Proceedings of the 2009 Symposium on Bio-inspired Learning and Intelligent Systems for Security

A research on current condition and problems of critical infrastructures and critical information infrastructures is described in this paper. This article includes issues, the discussion of problems and comments about each of the common Critical ...
Information Assurance in Critical Infrastructures via Wireless Sensor Networks
IAS '08: Proceedings of the 2008 The Fourth International Conference on Information Assurance and Security

Information assurance in critical infrastructure is an issue that has been addressed generally focusing on real-time or quasi real-time monitoring of the critical infrastructure; so that action could be undertaken when anomalies arise, to avoid more ...

Comments

Information & Contributors

Information

Published In

cover image Guide Proceedings

2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

Oct 2018

7818 pages

Copyright © 2018.

Publisher

IEEE Press

Publication History

Published: 01 October 2018

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

15
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 23 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Yuan ZGuo WJia JLi BSong DSalakhutdinov RKolter ZHeller KWeller AOliver NScarlett JBerkenkamp F(2024)SHINEProceedings of the 41st International Conference on Machine Learning10.5555/3692070.3694458(57887-57904)Online publication date: 21-Jul-2024
https://dl.acm.org/doi/10.5555/3692070.3694458
Mehrotra SDegachi CVereschak OJonker CTielman M(2024)A Systematic Review on Fostering Appropriate Trust in Human-AI Interaction: Trends, Opportunities and ChallengesACM Journal on Responsible Computing10.1145/36964491:4(1-45)Online publication date: 21-Sep-2024
https://dl.acm.org/doi/10.1145/3696449
Milani STopin NVeloso MFang F(2024)Explainable Reinforcement Learning: A Survey and Comparative ReviewACM Computing Surveys10.1145/361686456:7(1-36)Online publication date: 9-Apr-2024
https://dl.acm.org/doi/10.1145/3616864
Cheng ZWu XYu JSun WGuo WXing XOh ANaumann TGloberson ASaenko KHardt MLevine S(2023)StateMaskProceedings of the 37th International Conference on Neural Information Processing Systems10.5555/3666122.3668850(62457-62487)Online publication date: 10-Dec-2023
https://dl.acm.org/doi/10.5555/3666122.3668850
Yu JGuo WQin QWang GWang TXing XCalandrino JTroncoso C(2023)AIRSProceedings of the 32nd USENIX Conference on Security Symposium10.5555/3620237.3620650(7375-7392)Online publication date: 9-Aug-2023
https://dl.acm.org/doi/10.5555/3620237.3620650
Amitai YAgmon NAn BRicci AYeoh W(2023)Enhancing User Understanding of Reinforcement Learning Agents Through Visual ExplanationsProceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems10.5555/3545946.3599130(2937-2939)Online publication date: 30-May-2023
https://dl.acm.org/doi/10.5555/3545946.3599130
Huber TDemmler MMertes SOlson MAndré EAgmon NAn BRicci AYeoh W(2023)GANterfactual-RL: Understanding Reinforcement Learning Agents' Strategies through Visual Counterfactual ExplanationsProceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems10.5555/3545946.3598751(1097-1106)Online publication date: 30-May-2023
https://dl.acm.org/doi/10.5555/3545946.3598751
Mehrotra SJorge CJonker CTielman M(2023)Integrity-based Explanations for Fostering Appropriate Trust in AI AgentsACM Transactions on Interactive Intelligent Systems10.1145/361057814:1(1-36)Online publication date: 24-Jul-2023
https://dl.acm.org/doi/10.1145/3610578
Aldridge ABethel CCastellano GRiek LCakmak MLeite I(2023)M-OAT Shared Meta-Model Framework for Effective Collaborative Human-Autonomy TeamingCompanion of the 2023 ACM/IEEE International Conference on Human-Robot Interaction10.1145/3568294.3580169(663-666)Online publication date: 13-Mar-2023
https://dl.acm.org/doi/10.1145/3568294.3580169
Wischnewski MKrämer NMüller E(2023)Measuring and Understanding Trust Calibrations for Automated Systems: A Survey of the State-Of-The-Art and Future DirectionsProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581197(1-16)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581197
Show More Cited By

View Options

View options

Figures

Tables

Media

View Table of Conten