Abstract
Distributed robotic systems can benefit from automatic controller design and online adaptation by reinforcement learning (RL), but often suffer from the limitations of partial observability. In this paper, we address the twin problems of limited local experience and locally observed but not necessarily telling reward signals encountered in such systems.We combine direct search in policy space with an agreement algorithm to efficiently exchange local rewards and experience among agents. We demonstrate improved learning ability on the locomotion problem for self-reconfiguring modular robots in simulation, and show that a fully distributed implementation can learn good policies just as fast as the centralized implementation. Our results suggest that prior work on centralized RL algorithms for modular robots may be made effective in practice through the application of agreement algorithms. This approach could be fruitful in many cooperative situations, whenever robots need to learn similar behaviors, but have access only to local information.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Baxter, J., Bartlett, P.L.: Infinite-horizon gradient-based policy search. J. of Artificial Intelligence Res. 15, 319–350 (2001)
Bertsekas, D.P., Tsitsiklis, J.N.: Parallel and Distributed Computation: Numerical Methods. Athena Scientific (1997)
Chang, Y.-H., Ho, T., Kaelbling, L.P.: All learning is local: Multi-agent learning in global reward games. In: Advances in Neural Information Processing Systems, vol. 16 (2004)
Fernandez, F., Parker, L.E.: Learning in large cooperative multi-robot domains. Int. J. of Robotics and Automation 16(4), 217–226 (2001)
Guestrin, C., Koller, D., Parr, R.: Multiagent planning with factored MDPs. In: Advances in Neural Information Processing Systems, vol. 14 (2002)
Hu, J., Wellman, M.P.: Multiagent reinforcement learning: Theoretical framework and an algorithm. In: Proc. Int. Conf. on Machine Learning, pp. 242–250 (1998)
Kok, J.R., Vlassis, N.: Collaborative multiagent reinforcement learning by payoff propagation. J. of Machine Learning Res. 7, 1789–1828 (2006)
Lynch, K.M., Schwartz, I.B., Yang, P., Freeman, R.: Decentralized environmental modeling by mobile sensor networks. IEEE Trans. on Robotics 24(3), 710–724 (2008)
Matarić, M.J.: Reinforcement learning in the multi-robot domain. Autonomous Robots 4(1), 73–83 (1997)
Moallemi, C.C., Van Roy, B.: Distributed optimization in adaptive networks. In: Advances in Neural Information Processing Systems, vol. 15 (2003)
Moallemi, C.C., Van Roy, B.: Consensus propagation. IEEE Trans. on Information Theory 52(11) (2006)
Peshkin, L.: Reinforcement Learning by Policy Search. PhD thesis, Brown University (2001)
Schneider, J., Wong, W.-K., Moore, A., Riedmiller, M.: Distributed value functions. In: Proc. Int. Conf. on Machine Leanring (1999)
Tsitsiklis, J.N., Bertsekas, D.P., Athans, M.: Distributed asynchronous deterministic and stochastic gradient optimization algorithms. IEEE Trans. on Automatic Control AC-31(9), 803–812 (1986)
Varshavskaya, P., Kaelbling, L.P., Rus, D.: Distributed learning for modular robots. In: Proc. Int. Conf. on Robots and Systems (2004)
Varshavskaya, P., Kaelbling, L.P., Rus, D.: Automated design of adaptive controllers for modular robots using reinforcement learning. Int. J. of Robotics Res. 27(3–4), 505–526 (2008)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2009 Springer-Verlag Berlin Heidelberg
About this chapter
Cite this chapter
Varshavskaya, P., Kaelbling, L.P., Rus, D. (2009). Efficient Distributed Reinforcement Learning through Agreement. In: Asama, H., Kurokawa, H., Ota, J., Sekiyama, K. (eds) Distributed Autonomous Robotic Systems 8. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-00644-9_33
Download citation
DOI: https://doi.org/10.1007/978-3-642-00644-9_33
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-00643-2
Online ISBN: 978-3-642-00644-9
eBook Packages: EngineeringEngineering (R0)