The Role of Foundation Models in Neuro-Symbolic Learning and Reasoning
Pages 84 - 100
Abstract
Neuro-Symbolic AI (NeSy) holds promise to ensure the safe deployment of AI systems, as interpretable symbolic techniques provide formal behaviour guarantees. The challenge is how to effectively integrate neural and symbolic computation, to enable learning and reasoning from raw data. Existing pipelines that train the neural and symbolic components sequentially require extensive labelling, whereas end-to-end approaches are limited in terms of scalability, due to the combinatorial explosion in the symbol grounding problem. In this paper, we leverage the implicit knowledge within foundation models to enhance the performance in NeSy tasks, whilst reducing the amount of data labelling and manual engineering. We introduce a new architecture, called NeSyGPT, which fine-tunes a vision-language foundation model to extract symbolic features from raw data, before learning a highly expressive answer set program to solve a downstream task. Our comprehensive evaluation demonstrates that NeSyGPT has superior accuracy over various baselines, and can scale to complex NeSy tasks. Finally, we highlight the effective use of a large language model to generate the programmatic interface between the neural and symbolic components, significantly reducing the amount of manual engineering required. The Appendix is presented in the longer version of this paper, which contains additional results and analysis [8].
References
[1]
Aspis, Y., Broda, K., Lobo, J., Russo, A.: Embed2sym-scalable neuro-symbolic reasoning via clustered embeddings. In: Proceedings of the International Conference on Principles of Knowledge Representation and Reasoning, vol. 19, pp. 421–431 (2022)
[2]
Badreddine, S., d’Avila Garcez, A., Serafini, L., Spranger, M.: Logic tensor networks. Artif. Intell. 303, 103649 (2022).
[3]
Besold, T.R., et al.: Neural-symbolic learning and reasoning: a survey and interpretation. In: Neuro-Symbolic Artificial Intelligence: The State of the Art. IOS Press (2022)
[4]
Byerly A, Kalganova T, and Dear I No routing needed between capsules Neurocomputing 2021 463 545-553
[5]
Charalambous, T., Aspis, Y., Russo, A.: NeuralFastLAS: fast logic-based learning from raw data. arXiv preprint arXiv:2310.05145 (2023)
[6]
Cunnington D, Law M, Lobo J, and Russo A FFNSL: feed-forward neural-symbolic learner Mach. Learn. 2023 112 2 515-569
[7]
Cunnington, D., Law, M., Lobo, J., Russo, A.: Neuro-symbolic learning of answer set programs from raw data. In: Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, pp. 3586–3596 (2023)
[8]
Cunnington, D., Law, M., Lobo, J., Russo, A.: The role of foundation models in neuro-symbolic learning and reasoning. arXiv preprint arXiv:2402.01889 (2024)
[9]
Dai, W.Z., Muggleton, S.: Abductive knowledge induction from raw data. In: Zhou, Z.H. (ed.) Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pp. 1845–1851. International Joint Conferences on Artificial Intelligence Organization (2021).
[10]
Dai, W.Z., Xu, Q., Yu, Y., Zhou, Z.H.: Bridging machine learning and logical reasoning by abductive learning. Adv. Neural Inf. Process. Syst. 32 (2019)
[11]
Daniele, A., Campari, T., Malhotra, S., Serafini, L.: Deep symbolic learning: discovering symbols and rules from perceptions. In: Elkind, E. (ed.) Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI-23, pp. 3597–3605. International Joint Conferences on Artificial Intelligence Organization (2023).
[12]
Defresne, M., Barbe, S., Schiex, T.: Scalable coupling of deep learning with logical reasoning. In: Elkind, E. (ed.) Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI-23, pp. 3615–3623. International Joint Conferences on Artificial Intelligence Organization (2023).
[13]
Eiter T, Higuera N, Oetsch J, and Pritz M A neuro-symbolic ASP pipeline for visual question answering Theory Pract. Logic Program. 2022 22 5 739-754
[14]
Evans, R., et al.: Making sense of raw input. Artif. Intell. 299, 103521 (2021).
[15]
d’Avila Garcez, A., Gori, M., Lamb, L.C., Serafini, L., Spranger, M., Tran, S.N.: Neural-symbolic computing: an effective methodology for principled integration of machine learning and reasoning. FLAP 6(4), 611–632 (2019)
[16]
Gelfond M and Kahl Y Knowledge Representation, Reasoning, and the Design of Intelligent Agents: The Answer-set Programming Approach 2014 Cambridge, UK Cambridge University Press
[17]
Harnad S The symbol grounding problem Phys. D 1990 42 1–3 335-346
[18]
Hutchins D, Schlag I, Wu Y, Dyer E, and Neyshabur B Block-recurrent transformers Adv. Neural. Inf. Process. Syst. 2022 35 33248-33261
[19]
Ishay, A., Yang, Z., Lee, J.: Leveraging large language models to generate answer set programs. In: Proceedings of the 20th International Conference on Principles of Knowledge Representation and Reasoning, pp. 374–383 (2023).
[20]
Karp RM Miller RE, Thatcher JW, and Bohlinger JD Reducibility among combinatorial problems Complexity of Computer Computations 1972 Boston, MA, US Springer 85-103
[21]
Kassner, N., Schütze, H.: Negated and misprimed probes for pretrained language models: birds can talk, but cannot fly. arXiv preprint arXiv:1911.03343 (2019)
[22]
Law, M., Russo, A., Bertino, E., Broda, K., Lobo, J.: FastLAS: scalable inductive logic programming incorporating domain-specific optimisation criteria. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 2877–2885 (2020)
[23]
Law, M., Russo, A., Broda, K.: Logic-based learning of answer set programs. In: Reasoning Web, Explainable Artificial Intelligence - 15th International Summer School 2019, Bolzano, Italy, September 20-24, 2019, Tutorial Lectures, pp. 196–231 (2019)
[24]
Law, M., Russo, A., Broda, K.: The ilasp system for inductive learning of answer set programs. arXiv preprint arXiv:2005.00904 (2020)
[25]
Li, D., Li, J., Le, H., Wang, G., Savarese, S., Hoi, S.C.: Lavis: a library for language-vision intelligence. arXiv preprint arXiv:2209.09019 (2022)
[26]
Li, J., Li, D., Xiong, C., Hoi, S.: BLIP: bootstrapping language-image pre-training for unified vision-language understanding and generation. In: International Conference on Machine Learning, pp. 12888–12900. PMLR (2022)
[27]
Manhaeve, R., Dumancic, S., Kimmig, A., Demeester, T., De Raedt, L.: Deepproblog: neural probabilistic logic programming. Adv. Neural Inf. Process. Syst. 31 (2018)
[28]
Marconato, E., Teso, S., Vergari, A., Passerini, A.: Not all neuro-symbolic concepts are created equal: analysis and mitigation of reasoning shortcuts. Adv. Neural Inf. Process. Syst. 36 (2024)
[29]
Nye M, Tessler M, Tenenbaum J, and Lake BM Improving coherence and consistency in neural sequence models with dual-system, neuro-symbolic reasoning Adv. Neural. Inf. Process. Syst. 2021 34 25192-25204
[30]
OpenAI, et al.: GPT-4 technical report (2023)
[31]
Pryor, C., Dickens, C., Augustine, E., Albalak, A., Wang, W.Y., Getoor, L.: NeuPSL: neural probabilistic soft logic. In: Elkind, E. (ed.) Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI-23, pp. 4145–4153. International Joint Conferences on Artificial Intelligence Organization (2023).
[32]
Riegel, R., et al.: Logical neural networks. arXiv preprint arXiv:2006.13155 (2020)
[33]
Shindo H, Pfanschilling V, Dhami DS, and Kersting K ILP: thinking visual scenes as differentiable logic programs Mach. Learn. 2023 112 5 1465-1497
[34]
Singh, D., Jain, N., Jain, P., Kayal, P., Kumawat, S., Batra, N.: PlantDoc: a dataset for visual plant disease detection. In: Proceedings of the 7th ACM IKDD CoDS and 25th COMAD, pp. 249–253 (2020)
[35]
Skryagin, A., Ochs, D., Dhami, D.S., Kersting, K.: Scalable neural-probabilistic answer set programming. arXiv preprint arXiv:2306.08397 (2023)
[36]
Stammer, W., Schramowski, P., Kersting, K.: Right for the right concept: revising neuro-symbolic concepts by interacting with their explanations. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3619–3629 (2021)
[37]
Surís, D., Menon, S., Vondrick, C.: ViperGPT: visual inference via python execution for reasoning. arXiv preprint arXiv:2303.08128 (2023)
[38]
Wang, R., Zelikman, E., Poesia, G., Pu, Y., Haber, N., Goodman, N.D.: Hypothesis search: inductive reasoning with language models. arXiv preprint arXiv:2309.05660 (2023)
[39]
Yang, Z., Ishay, A., Lee, J.: NeurASP: embracing neural networks into answer set programming. In: Bessiere, C. (ed.) Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, pp. 1755–1762. International Joint Conferences on Artificial Intelligence Organization (2020).
[40]
Yang, Z., Ishay, A., Lee, J.: Coupling large language models with logic programming for robust and general reasoning from text. arXiv preprint arXiv:2307.07696 (2023)
[41]
Yujian L and Bo L A normalized Levenshtein distance metric IEEE Trans. Pattern Anal. Mach. Intell. 2007 29 6 1091-1095
Index Terms
- The Role of Foundation Models in Neuro-Symbolic Learning and Reasoning
Index terms have been assigned to the content through auto-classification.
Recommendations
Efficient symbolic policy learning with differentiable symbolic expression
NIPS '23: Proceedings of the 37th International Conference on Neural Information Processing SystemsDeep reinforcement learning (DRL) has led to a wide range of advances in sequential decision-making tasks. However, the complexity of neural network policies makes it difficult to understand and deploy with limited computational resources. Currently, ...
Symbolic assume-guarantee reasoning through BDD learning
ICSE 2014: Proceedings of the 36th International Conference on Software EngineeringBoth symbolic model checking and assume-guarantee reasoning aim to circumvent the state explosion problem. Symbolic model checking explores many states simultaneously and reports numerous erroneous traces. Automated assume-guarantee reasoning, on the ...
Deep symbolic learning: discovering symbols and rules from perceptions
IJCAI '23: Proceedings of the Thirty-Second International Joint Conference on Artificial IntelligenceNeuro-Symbolic (NeSy) integration combines symbolic reasoning with Neural Networks (NNs) for tasks requiring perception and reasoning. Most NeSy systems rely on continuous relaxation of logical knowledge, and no discrete decisions are made within the ...
Comments
Information & Contributors
Information
Published In
Sep 2024
440 pages
ISBN:978-3-031-71166-4
DOI:10.1007/978-3-031-71167-1
© The Author(s), under exclusive license to Springer Nature Switzerland AG 2024.
Publisher
Springer-Verlag
Berlin, Heidelberg
Publication History
Published: 10 September 2024
Author Tags
Qualifiers
- Article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 18 Feb 2025