research-article

Reinforcement Learning and Inverse Reinforcement Learning with System 1 and System 2

Author:

Alexander PeysakhovichAuthors Info & Claims

AIES '19: Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society

Pages 409 - 415

https://doi.org/10.1145/3306618.3314259

Published: 27 January 2019 Publication History

Abstract

Inferring a person's goal from their behavior is an important problem in applications of AI (e.g. automated assistants, recommender systems). The workhorse model for this task is the rational actor model - this amounts to assuming that people have stable reward functions, discount the future exponentially, and construct optimal plans. Under the rational actor assumption techniques such as inverse reinforcement learning (IRL) can be used to infer a person's goals from their actions. A competing model is the dual-system model. Here decisions are the result of an interplay between a fast, automatic, heuristic-based system 1 and a slower, deliberate, calculating system 2. We generalize the dual system framework to the case of Markov decision problems and show how to compute optimal plans for dual-system agents. We show that dual-system agents exhibit behaviors that are incompatible with rational actor assumption. We show that naive applications of rational-actor IRL to the behavior of dual-system agents can generate wrong inference about the agents' goals and suggest interventions that actually reduce the agent's overall utility. Finally, we adapt a simple IRL algorithm to correctly infer the goals of dual system decision-makers. This allows us to make interventions that help, rather than hinder, the dual-system agent's ability to reach their true goals.

References

[1]

George Ainslie. 2001. Breakdown of will .Cambridge University Press.

[2]

Susan Athey and Denis Nekipelov. 2010. A structural model of sponsored search advertising auctions. In Sixth ad auctions workshop, Vol. 15.

[3]

Adam Bear and David G Rand. 2016. Intuition, deliberation, and the evolution of cooperation. Proceedings of the National Academy of Sciences, Vol. 113, 4 (2016), 936--941.

[4]

Gharad Bryan, Dean Karlan, and Scott Nelson. 2010. Commitment devices. Annu. Rev. Econ., Vol. 2, 1 (2010), 671--698.

[5]

Ido Erev and Alvin E Roth. 1998. Predicting how people play games: Reinforcement learning in experimental games with unique, mixed strategy equilibria. American economic review (1998), 848--881.

[6]

Owain Evans, Andreas Stuhlmüller, and Noah D Goodman. 2016. Learning the Preferences of Ignorant, Inconsistent Agents. In AAAI. 323--329.

Digital Library

[7]

Drew Fudenberg and David K Levine. 2006. A dual-self model of impulse control. American economic review, Vol. 96, 5 (2006), 1449--1476.

[8]

Drew Fudenberg and David K Levine. 2012. Timing and self-control. Econometrica, Vol. 80, 1 (2012), 1--42.

[9]

Drew Fudenberg and Alexander Peysakhovich. 2016. Recency, records, and recaps: Learning and nonequilibrium behavior in a simple decision problem. ACM Transactions on Economics and Computation (TEAC), Vol. 4, 4 (2016), 23.

Digital Library

[10]

Joshua David Greene. 2014. Moral tribes: Emotion, reason, and the gap between us and them .Penguin.

[11]

Faruk Gul and Wolfgang Pesendorfer. 2001. Temptation and self-control. Econometrica, Vol. 69, 6 (2001), 1403--1435.

[12]

Dylan Hadfield-Menell, Stuart J Russell, Pieter Abbeel, and Anca Dragan. 2016. Cooperative inverse reinforcement learning. In Advances in neural information processing systems. 3909--3917.

Digital Library

[13]

Todd A Hare, Colin F Camerer, and Antonio Rangel. 2009. Self-control in decision-making involves modulation of the vmPFC valuation system. Science, Vol. 324, 5927 (2009), 646--648.

[14]

Ralph Hertwig, Greg Barron, Elke U Weber, and Ido Erev. 2004. Decisions from experience and the effect of rare events in risky choice. Psychological science, Vol. 15, 8 (2004), 534--539.

[15]

Ming Hsu, Meghana Bhatt, Ralph Adolphs, Daniel Tranel, and Colin F Camerer. 2005. Neural systems responding to degrees of uncertainty in human decision-making. Science, Vol. 310, 5754 (2005), 1680--1683.

[16]

Daniel Kahneman. 2011. Thinking, fast and slow. Vol. 1. Farrar, Straus and Giroux New York.

[17]

Sarah Keren, Avigdor Gal, and Erez Karpas. 2014. Goal Recognition Design. In ICAPS .

Digital Library

[18]

Jon Kleinberg and Sigal Oren. 2014. Time-inconsistent planning: a computational problem in behavioral economics. In Proceedings of the fifteenth ACM conference on Economics and computation. ACM, 547--564.

Digital Library

[19]

Jon Kleinberg, Sigal Oren, and Manish Raghavan. 2016. Planning problems for sophisticated agents with present bias. In Proceedings of the 2016 ACM Conference on Economics and Computation. ACM, 343--360.

Digital Library

[20]

Wouter Kool, Samuel J Gershman, and Fiery A Cushman. 2017. Cost-benefit arbitration between multiple reinforcement-learning systems. Psychological science, Vol. 28, 9 (2017), 1321--1333.

[21]

Wouter Kool, Samuel J Gershman, and Fiery A Cushman. 2018. Planning complexity registers as a cost in metacontrol. Journal of cognitive neuroscience (2018), 1--14.

Digital Library

[22]

Zeb Kurth-Nelson and A David Redish. 2009. Temporal-difference reinforcement learning with distributed representations. PLoS One, Vol. 4, 10 (2009), e7362.

[23]

Zeb Kurth-Nelson and A David Redish. 2010. A reinforcement learning model of precommitment in decision making. Frontiers in behavioral neuroscience, Vol. 4 (2010), 184.

[24]

David Laibson. 1997. Golden eggs and hyperbolic discounting. The Quarterly Journal of Economics, Vol. 112, 2 (1997), 443--478.

[25]

Joel Z Leibo, Vinicius Zambaldi, Marc Lanctot, Janusz Marecki, and Thore Graepel. 2017. Multi-agent reinforcement learning in sequential social dilemmas. In Proceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems. International Foundation for Autonomous Agents and Multiagent Systems, 464--473.

Digital Library

[26]

Adam Lerer and Alexander Peysakhovich. 2017. Maintaining cooperation in complex social dilemmas using deep reinforcement learning. arXiv preprint arXiv:1707.01068 (2017).

[27]

Ulrike Malmendier and Stefan Nagel. 2011. Depression babies: do macroeconomic experiences affect risk taking? The Quarterly Journal of Economics, Vol. 126, 1 (2011), 373--416.

[28]

Samuel M McClure, David I Laibson, George Loewenstein, and Jonathan D Cohen. 2004. Separate neural systems value immediate and delayed monetary rewards. Science, Vol. 306, 5695 (2004), 503--507.

[29]

Andrew Y Ng, Stuart J Russell, et al. 2000. Algorithms for inverse reinforcement learning. In ICML. 663--670.

Digital Library

[30]

Ted O'Donoghue and Matthew Rabin. 1999. Doing it now or later. American Economic Review, Vol. 89, 1 (1999), 103--124.

[31]

Alexander Peysakhovich. 2014. How to commit (if you must): Commitment contracts and the dual-self model. Journal of Economic Behavior & Organization, Vol. 101 (2014), 100--112.

[32]

Alexander Peysakhovich and Adam Lerer. 2018. Consequentialist conditional cooperation in social dilemmas with imperfect information. In ICLR .

[33]

Alexander Peysakhovich and David G Rand. 2015. Habits of virtue: Creating norms of cooperation and defection in the laboratory. Management Science, Vol. 62, 3 (2015), 631--647.

Digital Library

[34]

Deepak Ramachandran and Eyal Amir. {n. d.}. Bayesian inverse reinforcement learning. ({n. d.}).

[35]

David G Rand, Joshua D Greene, and Martin A Nowak. 2012. Spontaneous giving and calculated greed. Nature, Vol. 489, 7416 (2012), 427.

[36]

David G Rand, Alexander Peysakhovich, Gordon T Kraft-Todd, George E Newman, Owen Wurzbacher, Martin A Nowak, and Joshua D Greene. 2014. Social heuristics shape intuitive cooperation. Nature communications, Vol. 5 (2014), 3677.

[37]

Paul Resnick and Hal R Varian. 1997. Recommender systems. Commun. ACM, Vol. 40, 3 (1997), 56--59.

Digital Library

[38]

Amitai Shenhav, Matthew M Botvinick, and Jonathan D Cohen. 2013. The expected value of control: an integrative theory of anterior cingulate cortex function. Neuron, Vol. 79, 2 (2013), 217--240.

[39]

Rainer Storn and Kenneth Price. 1997. Differential evolution--a simple and efficient heuristic for global optimization over continuous spaces. Journal of global optimization, Vol. 11, 4 (1997), 341--359.

Digital Library

[40]

Richard S Sutton and Andrew G Barto. 1998. Introduction to reinforcement learning. Vol. 135. MIT press Cambridge.

Digital Library

[41]

Richard Thaler. 2012. The winner's curse: Paradoxes and anomalies of economic life .Simon and Schuster.

[42]

Richard H Thaler and Hersh M Shefrin. 1981. An economic theory of self-control. Journal of political Economy, Vol. 89, 2 (1981), 392--406.

[43]

Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, and Anind K Dey. 2008. Maximum Entropy Inverse Reinforcement Learning. In AAAI, Vol. 8. Chicago, IL, USA, 1433--1438.

Digital Library

Cited By

Liu SWang ZZhang YYang H(2024)Anomalous ride-hailing driver detection with deep transfer inverse reinforcement learningTransportation Research Part C: Emerging Technologies10.1016/j.trc.2023.104466159(104466)Online publication date: Feb-2024
https://doi.org/10.1016/j.trc.2023.104466
Benthall SShekman D(2023)Designing Fiduciary Artificial IntelligenceProceedings of the 3rd ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization10.1145/3617694.3623230(1-15)Online publication date: 30-Oct-2023
https://dl.acm.org/doi/10.1145/3617694.3623230
Gopinath DDeCastro JRosman GSumner EMorgan AHakimi SStent S(2022)HMIway-env: A Framework for Simulating Behaviors and Preferences to Support Human-AI Teaming in Driving2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)10.1109/CVPRW56347.2022.00480(4341-4349)Online publication date: Jun-2022
https://doi.org/10.1109/CVPRW56347.2022.00480
Show More Cited By

Index Terms

Reinforcement Learning and Inverse Reinforcement Learning with System 1 and System 2
1. Applied computing
  1. Law, social and behavioral sciences
    1. Economics
2. Theory of computation
  1. Theory and algorithms for application domains
    1. Machine learning theory
      1. Reinforcement learning

Recommendations

Reward Shaping in Episodic Reinforcement Learning
AAMAS '17: Proceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems

Recent advancements in reinforcement learning confirm that reinforcement learning techniques can solve large scale problems leading to high quality autonomous decision making. It is a matter of time until we will see large scale applications of ...
Model-based reinforcement learning: a computational model and an fMRI study

In this paper, we discuss an optimal decision-making problem in an unknown environment on the bases of both machine learning and brain learning. We present a model-based reinforcement learning (RL) in which the environment is directly estimated. Our RL ...
Multi-Agent Inverse Reinforcement Learning
ICMLA '10: Proceedings of the 2010 Ninth International Conference on Machine Learning and Applications

Learning the reward function of an agent by observing its behavior is termed inverse reinforcement learning and has applications in learning from demonstration or apprenticeship learning. We introduce the problem of multi-agent inverse reinforcement ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

AIES '19: Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society

January 2019

577 pages

ISBN:9781450363242

DOI:10.1145/3306618

General Chairs:
Vincent Conitzer
Duke University, USA
,
Gillian Hadfield
University of Toronto + Vector Institute, Canada; OpenAI, USA
,
Shannon Vallor
Santa Clara University, USA

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGAI: ACM Special Interest Group on Artificial Intelligence
AAAI: American Association for Artificial Intelligence

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 January 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

AIES '19

Sponsor:

SIGAI
AAAI

AIES '19: AAAI/ACM Conference on AI, Ethics, and Society

January 27 - 28, 2019

HI, Honolulu, USA

Acceptance Rates

Overall Acceptance Rate 61 of 162 submissions, 38%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
284
Total Downloads

Downloads (Last 12 months)27
Downloads (Last 6 weeks)2

Reflects downloads up to 25 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Liu SWang ZZhang YYang H(2024)Anomalous ride-hailing driver detection with deep transfer inverse reinforcement learningTransportation Research Part C: Emerging Technologies10.1016/j.trc.2023.104466159(104466)Online publication date: Feb-2024
https://doi.org/10.1016/j.trc.2023.104466
Benthall SShekman D(2023)Designing Fiduciary Artificial IntelligenceProceedings of the 3rd ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization10.1145/3617694.3623230(1-15)Online publication date: 30-Oct-2023
https://dl.acm.org/doi/10.1145/3617694.3623230
Gopinath DDeCastro JRosman GSumner EMorgan AHakimi SStent S(2022)HMIway-env: A Framework for Simulating Behaviors and Preferences to Support Human-AI Teaming in Driving2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)10.1109/CVPRW56347.2022.00480(4341-4349)Online publication date: Jun-2022
https://doi.org/10.1109/CVPRW56347.2022.00480
Cavalcante Siebert LLupetti MAizenberg EBeckers NZgonnikov AVeluwenkamp HAbbink DGiaccardi EHouben GJonker Cvan den Hoven JForster DLagendijk R(2022)Meaningful human control: actionable properties for AI system developmentAI and Ethics10.1007/s43681-022-00167-33:1(241-255)Online publication date: 18-May-2022
https://doi.org/10.1007/s43681-022-00167-3
Butlin PFourcade MKuipers BLazar SMulligan D(2021)AI Alignment and Human RewardProceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society10.1145/3461702.3462570(437-445)Online publication date: 21-Jul-2021
https://dl.acm.org/doi/10.1145/3461702.3462570
Arseniev DBaskakov DShkodyrev V(2019)Graph Model Approach to Hierarchy Control NetworkCyber-Physical Systems and Control10.1007/978-3-030-34983-7_20(195-211)Online publication date: 30-Nov-2019
https://doi.org/10.1007/978-3-030-34983-7_20
Arsenjev DBaskakov DShkodyrev V(2019)Distributed Ledger Technology and Cyber-Physical Systems. Multi-agent Systems. Concepts and TrendsComputational Science and Its Applications – ICCSA 201910.1007/978-3-030-24296-1_50(618-630)Online publication date: 29-Jun-2019
https://doi.org/10.1007/978-3-030-24296-1_50

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents