Abstract
Many benchmarks and challenges for AI and AGI exist, which help to reveal both short- and long-term topics and directions of research. We analyze elementary school Olympiad math tasks as a possible benchmark for AGI that can occupy a certain free niche capturing some limitations of the existing neural and symbolic systems better than other existing both language understanding and mathematical tests. A detailed comparison and analysis of implications of AGI is provided.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Hernández-Orallo, J., Minaya-Collado, N.: A formal definition of intelligence based on an intensional variant of Kolmogorov complexity. In: Proceedings of the International Symposium of Engineering of Intelligent Systems (EIS 1998), pp. 146–163. ICSC Press (1998)
Goertzel, B.: Artificial general intelligence: concept, state of the art, and future prospects. J. Artif. Gen. Intell. 5(1), 1–48 (2014)
Hernández-Orallo, J., MartÃnez-Plumed, F., Schmid, U., Siebers, M., Dowe, D.L.: Computer models solving intelligence test problems: progress and implications. Artif. Intell. 230, 74–107 (2016)
Agrawal, A., et al.: Don’t just assume; look and answer: overcoming priors for visual question answering. In: Proceedings of IEEE Conference on CVPR, pp. 4971–4980 (2018)
Johnson, J., et al.: CLEVR: a diagnostic dataset for compositional language and elementary visual reasoning. arXiv preprint arXiv:1612.06890 (2016)
Chesani, F., Mello, P., Milano, M.: Solving mathematical puzzles: a challenging competition for AI. AI Mag. 38(3), 83–94 (2017)
Ackerman, E.: Can winograd schemas replace turing test for defining human-level AI? IEEE Spectrum (2014)
Wang, A., et al.: GLUE: a multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461 (2018)
Clark, P.: Elementary school science and math tests as a driver for AI: take the Aristo challenge! In: Twenty-Seventh IAAI Conference (2015)
Clark, P., et al.: From ‘F’ to ‘A’ on the N.Y. regents science exams: an overview of the aristo project. arXiv preprint arXiv:1909.01958 (2019)
Wang, A.: SuperGLUE: a stickier benchmark for general-purpose language understanding systems. arXiv preprint arXiv:1905.00537 (2019)
Hudson, D.A., Manning, Ch.D.: GQA: a new dataset for real-world visual reasoning and compositional question answering. arXiv preprint arXiv:1902.09506 (2019)
Weitnauer, E., Ritter, H.: Physical bongard problems. In: Iliadis, L., Maglogiannis, I., Papadopoulos, H. (eds.) AIAI 2012. IAICT, vol. 381, pp. 157–163. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33409-2_17
Klenk, M., Forbus, K.: Analogical model formulation for transfer learning in AP physics. Artif. Intell. 173(18), 1615–1638 (2009)
Sloman, A.: Kantian philosophy of mathematics and young robots. In: Autexier, S., Campbell, J., Rubio, J., Sorge, V., Suzuki, M., Wiedijk, F. (eds.) CICM 2008. LNCS (LNAI), vol. 5144, pp. 558–573. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-85110-3_45
Saxton, D., Grefenstette, E., Hill, F., Kohli, P.: Analysing mathematical reasoning abilities of neural models. In: International Conference on Learning Representations (2019). https://openreview.net/forum?id=H1gR5iR5FX
Graves, A., et al.: Hybrid computing using a neural network with dynamic external memory. Nature 538(7626), 471–476 (2016)
Schlag, I., et al.: Enhancing the transformer with explicit relational encoding for math problem solving. arXiv preprint arXiv:1910.06611 (2019)
Seo, M., et al.: Solving geometry problems: combining text and diagram interpretation. In: Proceedings Conference on Empirical Methods in Natural Language Processing, pp. 1466–1476 (2015)
Hosseini, M., Hajishirzi, H., Etzioni, O., Kushman, N.: Learning to solve arithmetic word problems with verb categorization. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 523–533 (2014)
Hopkins, M., et al.: SemEval 2019 task 10: math question answering. In: Proceedings of the 13th International Workshop on Semantic Evaluation (SemEval-2019), pp. 893–899 (2019)
Mitra, A., Baral, C.: Learning to automatically solve logic grid puzzles. In: Proceedings Conference on Empirical Methods in Natural Language Processing, pp. 1023–1033 (2015)
Sachan, M., Xing, E.: Learning to solve geometry problems from natural language demonstrations in textbooks. In: Proceedings of the 6th Joint Conference on Lexical and Computational Semantics, pp. 251–261 (2017)
Marcus, G.: The next decade in AI: four steps towards robust artificial intelligence. arXiv:2002.06177 (2020)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Potapov, A. et al. (2020). Analyzing Elementary School Olympiad Math Tasks as a Benchmark for AGI. In: Goertzel, B., Panov, A., Potapov, A., Yampolskiy, R. (eds) Artificial General Intelligence. AGI 2020. Lecture Notes in Computer Science(), vol 12177. Springer, Cham. https://doi.org/10.1007/978-3-030-52152-3_29
Download citation
DOI: https://doi.org/10.1007/978-3-030-52152-3_29
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-52151-6
Online ISBN: 978-3-030-52152-3
eBook Packages: Computer ScienceComputer Science (R0)