Article

Skill acquisition via transfer learning and advice taking

Authors:

Richard MaclinAuthors Info & Claims

ECML'06: Proceedings of the 17th European conference on Machine Learning

Pages 425 - 436

https://doi.org/10.1007/11871842_41

Published: 18 September 2006 Publication History

Abstract

We describe a reinforcement learning system that transfers skills from a previously learned source task to a related target task. The system uses inductive logic programming to analyze experience in the source task, and transfers rules for when to take actions. The target task learner accepts these rules through an advice-taking algorithm, which allows learners to benefit from outside guidance that may be imperfect. Our system accepts a human-provided mapping, which specifies the similarities between the source and target tasks and may also include advice about the differences between them. Using three tasks in the RoboCup simulated soccer domain, we demonstrate that this system can speed up reinforcement learning substantially.

References

[1]

K. Driessens and S. Dzeroski. Integrating experimentation and guidance in relational reinforcement learning. In Proc. ICML, 2002.

Digital Library

[2]

G. Fung, S. Sandilya, and B. Rao. Rule extraction from linear support vector machines. In Proc. KDD, 2005.

Digital Library

[3]

G. Kuhlmann, P. Stone, R. Mooney, and J. Shavlik. Guiding a reinforcement learner with natural language advice: Initial results in RoboCup soccer. In AAAI Workshop on Supervisory Control of Learning and Adaptive Systems, 2004.

[4]

R. Maclin and J. Shavlik. Creating advice-taking reinforcement learners. Machine Learning, 22:251-281, 1996.

Digital Library

[5]

R. Maclin, J. Shavlik, L. Torrey, T. Walker, and E. Wild. Giving advice about preferred actions to reinforcement learners via knowledge-based kernel regression. In Proc. AAAI, 2005.

Digital Library

[6]

S. Muggleton and L. De Raedt. Inductive logic programming: Theory and methods. Journal of Logic Programming 19,20, pages 629-679, 1994.

[7]

S. Singh. Transfer of learning by composing solutions of elemental sequential tasks. Machine Learning 8 (3-4), pages 323-339, 1992.

Digital Library

[8]

A. Srinivasan. The Aleph manual. http://web.comlab.ox.ac.uk/oucl/research/ areas /machlearn/Aleph/aleph.html, 2001.

[9]

P. Stone and R. Sutton. Scaling reinforcement learning toward RoboCup soccer. In Proc. ICML, 2001.

Digital Library

[10]

R. Sun. Knowledge extraction from reinforcement learning. New Learning Paradigms in Soft Computing, pages 170-180, 2002.

Digital Library

[11]

R. Sutton. Learning to predict by the methods of temporal differences. Machine Learning 3, pages 9-44, 1988.

Digital Library

[12]

R. Sutton. Generalization in reinforcement learning: Successful examples using sparse coarse coding. In Proc. NIPS, 1996.

[13]

R. Sutton and A. Barto. Reinforcement Learning: An Introduction. MIT Press, 1998.

Digital Library

[14]

M. Taylor and P. Stone. Behavior transfer for value-function-based reinforcement learning. In Proc. AAMAS, 2005.

Digital Library

[15]

L. Torrey, T. Walker, J. Shavlik, and R. Maclin. Using advice to transfer knowledge acquired in one reinforcement learning task to another. In Proc. ECML, 2005.

Digital Library

Cited By

Saha ODasgupta P(2018)Experience Learning From Basic Patterns for Efficient Robot Navigation in Indoor EnvironmentsJournal of Intelligent and Robotic Systems10.5555/3288993.328900292:3-4(545-564)Online publication date: 1-Dec-2018
https://dl.acm.org/doi/10.5555/3288993.3289002
Song JGao YWang HAn BJonker CMarsella SThangarajah JTuyls K(2016)Measuring the Distance Between Finite Markov Decision ProcessesProceedings of the 2016 International Conference on Autonomous Agents & Multiagent Systems10.5555/2936924.2936994(468-476)Online publication date: 9-May-2016
https://dl.acm.org/doi/10.5555/2936924.2936994
Cuayáhuitl Hvan Otterlo MDethlefs NFrommberger L(2013)Machine learning for interactive systems and robotsProceedings of the 2nd Workshop on Machine Learning for Interactive Systems: Bridging the Gap Between Perception, Action and Communication10.1145/2493525.2493530(19-28)Online publication date: 4-Aug-2013
https://dl.acm.org/doi/10.1145/2493525.2493530
Show More Cited By

Index Terms

Skill acquisition via transfer learning and advice taking
1. Computing methodologies
  1. Machine learning

Recommendations

Knowledge transfer via advice taking
K-CAP '05: Proceedings of the 3rd international conference on Knowledge capture

We present a framework for knowledge transfer from one reinforcement learning task to a related task through advice-taking mechanisms. We discuss the importance of transfer in complex domains such as RoboCup soccer, and show how to use automatically ...
Using advice to transfer knowledge acquired in one reinforcement learning task to another
ECML'05: Proceedings of the 16th European conference on Machine Learning

We present a method for transferring knowledge learned in one task to a related task. Our problem solvers employ reinforcement learning to acquire a model for one task. We then transform that learned model into advice for a new task. A human teacher ...
Teachable reinforcement learning via advice distillation
NIPS '21: Proceedings of the 35th International Conference on Neural Information Processing Systems

Training automated agents to complete complex tasks in interactive environments is challenging: reinforcement learning requires careful hand-engineering of reward functions, imitation learning requires specialized infrastructure and access to a human ...

Comments

Information & Contributors

Information

Published In

cover image Guide Proceedings

ECML'06: Proceedings of the 17th European conference on Machine Learning

September 2006

848 pages

ISBN:354045375X

Editors:
Johannes Fürnkranz
Knowledge Engineering Group, Technische Universität Darmstadt
,
Tobias Scheffer
Knowledge Engineering Group, Max Planck Institute for Computer Science, Saarbrücken, Germany
,
Myra Spiliopoulou
Faculty of Computer Science, Otto-von-Guericke-University Magdeburg, Saarbrücken, Germany

Sponsors

Pascal
Deutsche Forschungsgemeinschaft
Gobierno de España-Ministerio de Ciencia e InnovaciTn
Strato AG: Strato AG
IBM: IBM

Publisher

Springer-Verlag

Berlin, Heidelberg

Publication History

Published: 18 September 2006

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

14
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 12 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Saha ODasgupta P(2018)Experience Learning From Basic Patterns for Efficient Robot Navigation in Indoor EnvironmentsJournal of Intelligent and Robotic Systems10.5555/3288993.328900292:3-4(545-564)Online publication date: 1-Dec-2018
https://dl.acm.org/doi/10.5555/3288993.3289002
Song JGao YWang HAn BJonker CMarsella SThangarajah JTuyls K(2016)Measuring the Distance Between Finite Markov Decision ProcessesProceedings of the 2016 International Conference on Autonomous Agents & Multiagent Systems10.5555/2936924.2936994(468-476)Online publication date: 9-May-2016
https://dl.acm.org/doi/10.5555/2936924.2936994
Cuayáhuitl Hvan Otterlo MDethlefs NFrommberger L(2013)Machine learning for interactive systems and robotsProceedings of the 2nd Workshop on Machine Learning for Interactive Systems: Bridging the Gap Between Perception, Action and Communication10.1145/2493525.2493530(19-28)Online publication date: 4-Aug-2013
https://dl.acm.org/doi/10.1145/2493525.2493530
Stracuzzi DFern AAli KHess RPinto JLi NKönik TShapiro D(2011)An Application of Transfer to American FootballAI Magazine10.1609/aimag.v32i2.233632:2(107-125)Online publication date: 1-Jun-2011
https://dl.acm.org/doi/10.1609/aimag.v32i2.2336
Karpov IValsalam VMiikkulainen RLanzi P(2011)Human-assisted neuroevolution through shaping, advice and examplesProceedings of the 13th annual conference on Genetic and evolutionary computation10.1145/2001576.2001628(371-378)Online publication date: 12-Jul-2011
https://dl.acm.org/doi/10.1145/2001576.2001628
Boutsioukis GPartalas IVlahavas I(2011)Transfer learning in multi-agent reinforcement learning domainsProceedings of the 9th European conference on Recent Advances in Reinforcement Learning10.1007/978-3-642-29946-9_25(249-260)Online publication date: 9-Sep-2011
https://dl.acm.org/doi/10.1007/978-3-642-29946-9_25
Chatzidimitriou KPartalas IMitkas PVlahavas I(2011)Transferring evolved reservoir features in reinforcement learning tasksProceedings of the 9th European conference on Recent Advances in Reinforcement Learning10.1007/978-3-642-29946-9_22(213-224)Online publication date: 9-Sep-2011
https://dl.acm.org/doi/10.1007/978-3-642-29946-9_22
Illobre AGonzalez JOtero RSantos J(2010)Learning action descriptions of opponent behaviour in the robocup 2D simulation environmentProceedings of the 20th international conference on Inductive logic programming10.5555/2022735.2022751(105-113)Online publication date: 27-Jun-2010
https://dl.acm.org/doi/10.5555/2022735.2022751
Taylor MStone P(2009)Transfer Learning for Reinforcement Learning Domains: A SurveyThe Journal of Machine Learning Research10.5555/1577069.175583910(1633-1685)Online publication date: 1-Dec-2009
https://dl.acm.org/doi/10.5555/1577069.1755839
Torrey LShavlik JWalker TMaclin R(2007)Relational macros for transfer in reinforcement learningProceedings of the 17th international conference on Inductive logic programming10.5555/1793494.1793522(254-268)Online publication date: 19-Jun-2007
https://dl.acm.org/doi/10.5555/1793494.1793522
Show More Cited By

View Options

View options

Media

Figures

Other

Tables

View Table of Contents