Abstract
Benchmarking robotic technologies is of utmost importance for actual deployment of robotic applications in industrial and every-day environments, therefore many efforts have recently focused on this problem. Among the many different ways of benchmarking robotic systems, scientific competitions are recognized as one of the most effective ways of rapid development of scientific progress in a field. The RoboCup@Home league targets the development and deployment of autonomous service and assistive robot technology, being essential for future personal domestic applications, and offers an important approach to benchmarking domestic and service robots.
In this paper we present the new methodology for benchmarking DSR adopted in RoboCup@Home, that includes the definition of multiple benchmarks (tests) and of performance metrics based on the relationships between key abilities required to the robots and the tests. We also discuss the results of our benchmarking approach over the past years and provide an outlook on short- and mid-term goals of Home and of DSR in general.
Chapter PDF
Similar content being viewed by others
Keywords
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
References
van der Zant, T., Wisspeintner, T.: RoboCup@Home: Creating and Benchmarking Tomorrows Service Robot Applications. In: Robotic Soccer, pp. 521–528. I-Tech Education and Publishing (2007)
del Pobil, A.: Why do We Need Benchmarks in Robotics Research? In: Proc. of the Workshop on Benchmarks in Robotics Research, IEEE/RSJ International Conference on Intelligent Robots and Systems (2006)
Sabanovic, S., Michalowski, M., Simmons, R.: Robots in the wild: observing human-robot social interaction outside the lab. In: 9th IEEE International Workshop on Advanced Motion Control, pp. 596–601 (2006)
Balch, T., Yanco, H.A.: Ten years of the aaai mobile robot competition and exhibition: looking back and to the future. AI Magazine 23(1), 13–22 (2002)
Kitano, H., Asada, M., Kuniyoshi, Y., Noda, I., Osawa, E., Matsubara, H.: RoboCup: A Challenge Problem for AI. AI Magazine 18(1), 73–85 (1997)
The RoboCup Federation: RoboCup (2008), http://www.robocup.org/ (Retrieved 09/2008)
Meeden, L., Schultz, A.C., Balch, T.R., Bhargava, R., Haigh, K.Z., Bohlen, M., Stein, C., Miller, D.P.: The aaai 1999 mobile robot competitions and exhibitions. AI Magazine 21(3), 69–78 (2000)
Kitano, H., Tadokoro, S.: RoboCup Rescue: A Grand Challenge for Multiagent and Intelligent Systems. AI Magazine 22(1), 39–52 (2001)
Steinfeld, A., Fong, T., Kaber, D.B., Lewis, M., Scholtz, J., Schultz, A.C., Goodrich, M.A.: Common metrics for human-robot interaction. In: Proc. of HRI, pp. 33–40 (2006)
Yanco, H.A., Drury, J.L., Scholtz, J.: Beyond Usability Evaluation: Analysis of Human-Robot Interaction at a Major Robotics Competition. Human-Computer Interaction 19, 117–149 (2004)
Drury, J.L., Yanco, H.A., Scholtz, J.: Using competitions to study human-robot interaction in urban search and rescue. Interactions 12(2), 39–41 (2005)
Prassler, E., Hägele, M., Siegwart, R.: International Contest for Cleaning Robots: Fun Event or a First Step towards Benchmarking Service Robots (2006)
Savage, J., Ayala, F., Cuellar, S., Weitzenfeld, A.: The use of scripts based on conceptual dependency primitives for the operation of service mobile robots. In: Proceedings of the International RoboCup Symposium 2008 (CD-ROM Proceedings) (2008)
Ruiz-del-Solar, J.: Personal robots as ubiquitous-multimedial-mobile web interfaces. In: Proc. of 5th Latin American Web Congress (LA-WEB), pp. 120–127 (2007)
Schiffer, S., Ferrein, A., Lakemeyer, G.: Football is coming Home. In: Proc. of International Symposium on Practical Cognitive Agents and Robots (PCAR 2006). University of Western Australia Press (2006)
Doostdar, M., Schiffer, S., Lakemeyer, G.: A Robust Speech Recognition System for Service-Robotics Applications. In: Iocchi, L., Matsubara, H., Weitzenfeld, A., Zhou, C. (eds.) RoboCup 2008. LNCS (LNAI), vol. 5399, pp. 1–12. Springer, Heidelberg (2009)
Correa, M., Ruiz-del-Solar, J., Bernuy, F.: Face recognition for human-robot interaction applications: A comparative study. In: Proceedings of the International RoboCup Symposium 2008 (CD-ROM Proceedings) (2008)
Knox, W.B., Lee, J., Stone, P.: Domestic interaction on a segway base. In: Proceedings of the International RoboCup Symposium 2008 (CD-ROM Proceedings) (2008)
Belle, V., Deselaers, T., Schiffer, S.: Randomized trees for real-time one-step face detection and recognition. In: Proceedings of the 19th International Conference on Pattern Recognition (ICPR 2008), December 8-11. IEEE Computer Society, Los Alamitos (2008)
Loncomilla, P., Ruiz-del-Solar, J.: Robust object recognition using wide baseline matching for robocup applications. In: Visser, U., Ribeiro, F., Ohashi, T., Dellaert, F. (eds.) RoboCup 2007. LNCS (LNAI), vol. 5001, pp. 441–448. Springer, Heidelberg (2008)
Francke, H., Ruiz-del-Solar, J., Verschae, R.: Real-time hand gesture detection and recognition using boosted classifiers and active learning. In: Mery, D., Rueda, L. (eds.) PSIVT 2007. LNCS, vol. 4872, pp. 533–547. Springer, Heidelberg (2007)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2010 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Wisspeintner, T., van der Zan, T., Iocchi, L., Schiffer, S. (2010). RoboCup@Home: Results in Benchmarking Domestic Service Robots. In: Baltes, J., Lagoudakis, M.G., Naruse, T., Ghidary, S.S. (eds) RoboCup 2009: Robot Soccer World Cup XIII. RoboCup 2009. Lecture Notes in Computer Science(), vol 5949. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-11876-0_34
Download citation
DOI: https://doi.org/10.1007/978-3-642-11876-0_34
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-11875-3
Online ISBN: 978-3-642-11876-0
eBook Packages: Computer ScienceComputer Science (R0)