research-article

Towards adaptive programming: integrating reinforcement learning into a programming language

Authors:

Christopher Simpkins,

Charles Isbell, Jr.,

Michael MateasAuthors Info & Claims

OOPSLA '08: Proceedings of the 23rd ACM SIGPLAN conference on Object-oriented programming systems languages and applications

Pages 603 - 614

https://doi.org/10.1145/1449764.1449811

Published: 19 October 2008 Publication History

Abstract

Current programming languages and software engineering paradigms are proving insufficient for building intelligent multi-agent systems--such as interactive games and narratives--where developers are called upon to write increasingly complex behavior for agents in dynamic environments. A promising solution is to build adaptive systems; that is, to develop software written specifically to adapt to its environment by changing its behavior in response to what it observes in the world. In this paper we describe a new programming language, An Adaptive Behavior Language (A2BL), that implements adaptive programming primitives to support partial programming, a paradigm in which a programmer need only specify the details of behavior known at code-writing time, leaving the run-time system to learn the rest. Partial programming enables programmers to more easily encode software agents that are difficult to write in existing languages that do not offer language-level support for adaptivity. We motivate the use of partial programming with an example agent coded in a cutting-edge, but non-adaptive agent programming language (ABL), and show how A2BL can encode the same agent much more naturally.

References

[1]

David Andre and Stuart Russell. Programmable reinforcement learning agents. In Advances in Neural Information Processing Systems, volume 13, 2001.

[2]

David Andre and Stuart Russell. State abstraction for programmable reinforcement learning agents. In AAAI-02, Edmonton, Alberta, 2002. AAAI Press.

Digital Library

[3]

Sooraj Bhat, Charles Isbell, and Michael Mateas. On the difficulty of modular reinforcement learning for real-world partial programming. In Proceedings of the Twenty-First National Conference on Artificial Intelligence (AAAI-06), Boston, MA, USA, July 2006.

Digital Library

[4]

Thomas G. Dietterich. The MAXQ method for hierarchical reinforcement learning. In Proc. 15th International Conf. on Machine Learning, pages 118--126. Morgan Kaufmann, San Francisco, CA, 1998.

Digital Library

[5]

Leslie Pack Kaelbling, Michael L. Littman, and Andrew P. Moore. Reinforcement learning: A survey. Journal of Artificial Intelligence Research}, 237--285, 1996.

Digital Library

[6]

A. B. Loyall and J. Bates. Hap: A reactive adaptive architecture for agents. Technical Report CMU-CS-91-147, 1991.

[7]

Michael Mateas and Andrew Stern. Facade: An experiment in building a fully-realized interactive drama. In Game Developers Conference: Game Design Track, San Jose, CA, March 2003.

[8]

Michael Mateas and Andrew Stern. Life-like Characters. Tools, Affective Functions and Applications, chapter A Behavior Language: Joint Action and Behavioral Idioms. Springer, 2004.

[9]

Tom Mitchell. Machine Learning. McGraw-Hill, 1997.

Digital Library

[10]

Peter Norvig. Decision theory: The language of adaptive agent software. Presentation, March 1998. http://www.norvig.com/adaptive/index.htm

[11]

Peter Norvig and David Cohn. Adaptive software, 1998. http://norvig.com/adapaper-pcai.html

Digital Library

[12]

Ronald Parr and Stuart Russell. Reinforcement learning with hierarchies of machines. In Michael I. Jordan, Michael J. Kearns, and Sara A. Solla, editors, Advances in Neural Information Processing Systems, volume 10. The MIT Press, 1998.

Digital Library

[13]

Stuart Russell and Peter Norvig. Artificial Intelligence: A Modern Approach. Prenticce Hall, Upper Saddle River, NJ, 2003.

Digital Library

[14]

R.S. Sutton and A.G. Barto. Reinforcement Learning: An Introduction. MIT Press, Cambridge, MA, 1998.

Digital Library

[15]

Sprague, N., and Ballard, D. Multiple-Goal Reinforcement Learning with Modular Sarsa(0). In Proceedings of the Eighteenth International Joint Conference on Artificial Intelligence, 2003. Workshop paper.

Digital Library

Cited By

Zhou HMin HLin Y(2019)An Autonomous Task Algorithm Based on Behavior Trees for Robot2019 2nd China Symposium on Cognitive Computing and Hybrid Intelligence (CCHI)10.1109/CCHI.2019.8901959(64-70)Online publication date: Sep-2019
https://doi.org/10.1109/CCHI.2019.8901959
Li D(2012)A Kind of Adaptive Program Design MethodInternational Journal of Information and Education Technology10.7763/IJIET.2012.V2.128(274-277)Online publication date: 2012
https://doi.org/10.7763/IJIET.2012.V2.128
De Raedt LNijssen S(2011)Towards programming languages for machine learning and data miningProceedings of the 19th international conference on Foundations of intelligent systems10.5555/2029759.2029763(25-32)Online publication date: 28-Jun-2011
https://dl.acm.org/doi/10.5555/2029759.2029763
Show More Cited By

Index Terms

Towards adaptive programming: integrating reinforcement learning into a programming language
1. Software and its engineering
  1. Software notations and tools
    1. General programming languages
      1. Language features

Recommendations

Towards adaptive programming: integrating reinforcement learning into a programming language

Current programming languages and software engineering paradigms are proving insufficient for building intelligent multi-agent systems--such as interactive games and narratives--where developers are called upon to write increasingly complex behavior for ...
What Is Object-Oriented Programming?

The meaning of the term 'object oriented' is examined in the context of the general-purpose programming language C++. This choice is made partly to introduce C++ and partly because C++ is one of the few languages that supports data abstraction, object-...
Towards a 3D Virtual Game for Learning Object-Oriented Programming Fundamentals and C++ Language
CSEDU 2015: Proceedings of the 7th International Conference on Computer Supported Education - Volume 2

Object-Oriented Programming (OOP) paradigm is one of the most common paradigm in introductory programming courses. However, novices often have difficulties to understand the basic concepts which are of a high level of abstraction. Either tangible and ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

OOPSLA '08: Proceedings of the 23rd ACM SIGPLAN conference on Object-oriented programming systems languages and applications

October 2008

654 pages

ISBN:9781605582153

DOI:10.1145/1449764

General Chair:
Gail E. Harris
Instantiated Software Inc.
,
Program Chairs:
Gregor Kiczales
University of British Columbia
,
Dirk Riehle
SAP Research
,
Andrew P. Black
Portland State University

ACM SIGPLAN Notices Volume 43, Issue 10
September 2008
613 pages
ISSN:0362-1340
EISSN:1558-1160
DOI:10.1145/1449955
Issue’s Table of Contents

Copyright © 2008 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 19 October 2008

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

OOPSLA08

Sponsor:

OOPSLA08: ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications

October 19 - 23, 2008

TN, Nashville, USA

Acceptance Rates

Overall Acceptance Rate 268 of 1,244 submissions, 22%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

15
Total Citations
View Citations
628
Total Downloads

Downloads (Last 12 months)13
Downloads (Last 6 weeks)2

Reflects downloads up to 10 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Zhou HMin HLin Y(2019)An Autonomous Task Algorithm Based on Behavior Trees for Robot2019 2nd China Symposium on Cognitive Computing and Hybrid Intelligence (CCHI)10.1109/CCHI.2019.8901959(64-70)Online publication date: Sep-2019
https://doi.org/10.1109/CCHI.2019.8901959
Li D(2012)A Kind of Adaptive Program Design MethodInternational Journal of Information and Education Technology10.7763/IJIET.2012.V2.128(274-277)Online publication date: 2012
https://doi.org/10.7763/IJIET.2012.V2.128
De Raedt LNijssen S(2011)Towards programming languages for machine learning and data miningProceedings of the 19th international conference on Foundations of intelligent systems10.5555/2029759.2029763(25-32)Online publication date: 28-Jun-2011
https://dl.acm.org/doi/10.5555/2029759.2029763
Bauer TErwig MFern APinto JKhoo SSiek J(2011)Adaptation-based programming in javaProceedings of the 20th ACM SIGPLAN workshop on Partial evaluation and program manipulation10.1145/1929501.1929518(81-90)Online publication date: 24-Jan-2011
https://dl.acm.org/doi/10.1145/1929501.1929518
Eitan NHarel D(2011)Adaptive Behavioral ProgrammingProceedings of the 2011 IEEE 23rd International Conference on Tools with Artificial Intelligence10.1109/ICTAI.2011.109(685-692)Online publication date: 7-Nov-2011
https://dl.acm.org/doi/10.1109/ICTAI.2011.109
De Raedt LNijssen S(2011)Towards Programming Languages for Machine Learning and Data Mining (Extended Abstract)Foundations of Intelligent Systems10.1007/978-3-642-21916-0_3(25-32)Online publication date: 2011
https://doi.org/10.1007/978-3-642-21916-0_3
Pinto JFern ABauer TErwig M(2010)Robust Learning for Adaptive Programs by Leveraging Program StructureProceedings of the 2010 Ninth International Conference on Machine Learning and Applications10.1109/ICMLA.2010.150(943-948)Online publication date: 12-Dec-2010
https://dl.acm.org/doi/10.1109/ICMLA.2010.150
Rousi R(2022)Will Robots Know That They Are Robots? The Ethics of Utilizing Learning MachinesCulture and Computing10.1007/978-3-031-05434-1_31(464-476)Online publication date: 16-Jun-2022
https://doi.org/10.1007/978-3-031-05434-1_31
Patra SMason JGhallab MNau DTraverso P(2021)Deliberative Acting, Planning and Learning with Hierarchical Operational ModelsArtificial Intelligence10.1016/j.artint.2021.103523(103523)Online publication date: May-2021
https://doi.org/10.1016/j.artint.2021.103523
Petrovska APretschner A(2019)Learning Approach for Smart Self-Adaptive Cyber-Physical Systems2019 IEEE 4th International Workshops on Foundations and Applications of Self* Systems (FAS*W)10.1109/FAS-W.2019.00061(234-236)Online publication date: Jun-2019
https://doi.org/10.1109/FAS-W.2019.00061
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents