From Deep Learning to Deep Reasoning

The rise of big data and big compute has brought modern neural networks to many walks of digital life, thanks to the relative ease of constructing large models that scale to the real world. Current successes of Transformers and self-supervised pretraining on massive data have led some to believe that deep neural networks will be able to do almost everything once we have sufficient data and computational resources. However, neural networks are fast to exploit surface statistics but fail miserably to generalize to novel combinations. This is because they are not designed for deliberate reasoning -- the capacity to deliberately deduce new knowledge out of the contextualized data. This tutorial reviews recent developments to extend the capacity of neural networks to "learning-to-reason'' from data, where the task is to determine if the data entails a conclusion. This capacity opens up new ways to generate insights from data through arbitrary compositional querying without the need of predefining a narrow set of tasks. The tutorial consists of four parts. The first part covers the learning-to-reason framework, and explains how neural networks can serve as a strong backbone for reasoning through its natural operations such as binding, attention & dynamic computational graphs. The second part goes into more detail on how neural networks perform reasoning over unstructured and structured data, and across modalities. The third part reviews neural memories and their role in reasoning. The last part discusses generalization to novel combinations, under less supervision and with more knowledge.