Why Does Deep and Cheap Learning Work So Well?

Abstract

We show how the success of deep learning could depend not only on mathematics but also on physics: although well-known mathematical theorems guarantee that neural networks can approximate arbitrary functions well, the class of functions of practical interest can frequently be approximated through “cheap learning” with exponentially fewer parameters than generic ones. We explore how properties frequently encountered in physics such as symmetry, locality, compositionality, and polynomial log-probability translate into exceptionally simple neural networks. We further argue that when the statistical process generating the data is of a certain hierarchical form prevalent in physics and machine learning, a deep neural network can be more efficient than a shallow one. We formalize these claims using information theory and discuss the relation to the renormalization group. We prove various “no-flattening theorems” showing when efficient linear deep networks cannot be accurately approximated by shallow ones without efficiency loss; for example, we show that n variables cannot be multiplied using fewer than $2^n$ neurons in a single hidden layer.

Appendix A: The Polynomial No-Flattening Theorem

We saw above that a neural network can compute polynomials accurately and efficiently at linear cost, using only about 4 neurons per multiplication. For example, if n is a power of two, then the monomial $\prod _{i=1}^n x_i$ can be evaluated using 4n neurons arranged in a binary tree network with $\log _2 n$ hidden layers. In this appendix, we will prove a no-flattening theorem demonstrating that flattening polynomials is exponentially expensive.

Theorem

Suppose we are using a generic smooth activation function $\sigma (x) = \sum _{k=0}^\infty \sigma _k x^k$, where $\sigma _k \ne 0$ for $0\le k\le n$. Then for any desired accuracy $\epsilon >0$, there exists a neural network that can implement the function $\prod _{i=1}^n x_i$ using a single hidden layer of $2^n$ neurons. Furthermore, this is the smallest possible number of neurons in any such network with only a single hidden layer.

This result may be compared to problems in Boolean circuit complexity, notably the question of whether $TC^0 = TC^1$ [50]. Here circuit depth is analogous to number of layers, and the number of gates is analogous to the number of neurons. In both the Boolean circuit model and the neural network model, one is allowed to use neurons/gates which have an unlimited number of inputs. The constraint in the definition of $TC^i$ that each of the gate elements be from a standard universal library (AND, OR, NOT, Majority) is analogous to our constraint to use a particular nonlinear function. Note, however, that our theorem is weaker by applying only to depth 1, while $TC^0$ includes all circuits of depth O(1).

1.1 A.1 Proof that $2^n$ Neurons are Sufficient

A neural network with a single hidden layer of m neurons that approximates a product gate for n inputs can be formally written as a choice of constants $a_{ij}$ and $w_j$ satisfying

$$\begin{aligned} \sum _{j=1}^m w_j \sigma \left( \sum _{i=1}^n a_{ij} x_i\right) \approx \prod _{i=1}^n x_i. \end{aligned}$$

(A1)

Here, we use $\approx$ to denote that the two sides of (A1) have identical Taylor expansions up to terms of degree n; as we discussed earlier in our construction of a product gate for two inputs, this exables us to achieve arbitrary accuracy $\epsilon$ by first scaling down the factors $x_i$, then approximately multiplying them and finally scaling up the result.

We may expand (A1) using the definition $\sigma (x) = \sum _{k=0}^\infty \sigma _k x^k$ and drop terms of the Taylor expansion with degree greater than n, since they do not affect the approximation. Thus, we wish to find the minimal m such that there exist constants $a_{ij}$ and $w_j$ satisfying

$$\begin{aligned} \sigma _n\sum _{j=1}^m w_j\left( \sum _{i=1}^n a_{ij} x_i\right) ^n= & {} \prod _{i=1}^n x_i, \end{aligned}$$

(A2)

$$\begin{aligned} \sigma _k\sum _{j=1}^m w_j\left( \sum _{i=1}^n a_{ij} x_i\right) ^k= & {} 0, \end{aligned}$$

(A3)

for all $0\le k \le n-1$. Let us set $m = 2^n$, and enumerate the subsets of $\{1,\ldots ,n\}$ as $S_1,\ldots ,S_m$ in some order. Define a network of m neurons in a single hidden layer by setting $a_{ij}$ equal to the function $s_i(S_j)$ which is $-1$ if $i\in S_j$ and $+1$ otherwise, setting

$$\begin{aligned} w_j\equiv {1\over 2^n n!\sigma _n}\prod _{i=1}^n a_{ij} = {(-1)^{|S_j|}\over 2^n n!\sigma _n}. \end{aligned}$$

(A4)

In other words, up to an overall normalization constant, all coefficients $a_{ij}$ and $w_j$ equal $\pm 1$, and each weight $w_j$ is simply the product of the corresponding $a_{ij}$.

We must prove that this network indeed satisfies Eqs. (A2) and (A3). The essence of our proof will be to expand the left hand side of Eq. (A1) and show that all monomial terms except $x_1\cdot \cdot \cdot x_n$ come in pairs that cancel. To show this, consider a single monomial $p(\mathbf x ) = x_1^{r_1}\cdots x_n^{r_n}$ where $r_1 + \ldots + r_n = r \le n$.

If $p(\mathbf x ) \ne \prod _{i=1}^n x_i$, then we must show that the coefficient of $p(\mathbf x )$ in $\sigma _r\sum _{j=1}^m w_j\left( \sum _{i=1}^n a_{ij} x_i\right) ^r$ is 0. Since $p(\mathbf x ) \ne \prod _{i=1}^n x_i$, there must be some $i_0$ such that $r_{i_0} = 0$. In other words, $p(\mathbf x )$ does not depend on the variable $x_{i_0}$. Since the sum in Eq. (A1) is over all combinations of ± signs for all variables, every term will be canceled by another term where the (non-present) $x_{i_0}$ has the opposite sign and the weight $w_j$ has the opposite sign:

$$\begin{aligned}&\sigma _r\sum _{j=1}^m w_j \left( \sum _{i=1}^n a_{ij} x_i\right) ^r \\&= \sigma _r\sum _{S_j} {(-1)^{|S_j|}\over 2^n n!\sigma _r}\left( \sum _{i=1}^n s_i(S_j) x_i\right) ^r \\&= \sigma _r\sum _{S_j\not \ni i_0} \Bigg [{(-1)^{|S_j|}\over 2^n n!\sigma _r}\left( \sum _{i=1}^n s_i(S_j) x_i\right) ^r\\&\quad + {(-1)^{|S_j \cup \{i_0\}|}\over 2^n n!\sigma _r}\left( \sum _{i=1}^n s_i(S_j \cup \{i_0\}) x_i\right) ^r\Bigg ] \\&= \sum _{S_j\not \ni i_0} {(-1)^{|S_j|}\over 2^n n!}\Bigg [\left( \sum _{i=1}^n s_i(S_j) x_i\right) ^r \\&\quad -\left( \sum _{i=1}^n s_i(S_j \cup \{i_0\}) x_i\right) ^r\Bigg ] \end{aligned}$$

Observe that the coefficient of $p(\mathbf x )$ is equal in $\left( \sum _{i=1}^n s_i(S_j) x_i\right) ^r$ and $\left( \sum _{i=1}^n s_i(S_j \cup \{i_0\}) x_i\right) ^r$, since $r_{i_0}=0$. Therefore, the overall coefficient of $p(\mathbf x )$ in the above expression must vanish, which implies that (A3) is satisfied.

If instead $p(\mathbf x ) = \prod _{i=1}^n x_i$, then all terms have the coefficient of $p(\mathbf x )$ in $\left( \sum _{i=1}^n a_{ij} x_i\right) ^n$ is $n!\, \prod _{i=1}^n a_{ij} = (-1)^{|S_j|} n!$, because all n! terms are identical and there is no cancelation. Hence, the coefficient of $p(\mathbf x )$ on the left-hand side of (A2) is

$$\begin{aligned} \sigma _n\sum _{j=1}^m \frac{(-1)^{|S_j|}}{2^n n! \sigma _n} (-1)^{|S_j|} n! = 1, \end{aligned}$$

completing our proof that this network indeed approximates the desired product gate.

From the standpoint of group theory, our construction involves a representation of the group $G = \mathbb {Z}_2^n$, acting upon the space of polynomials in the variables $x_1, x_2, \ldots , x_n$. The group G is generated by elements $g_i$ such that $g_i$ flips the sign of $x_i$ wherever it occurs. Then, our construction corresponds to the computation

$$\begin{aligned} \mathbf f (x_1,\ldots ,x_n) = (1 - g_1)(1 - g_2) \cdots (1-g_n)\sigma (x_1+x_2+\ldots +x_n). \end{aligned}$$

Every monomial of degree at most n, with the exception of the product $x_1\cdots x_n$, is sent to 0 by $(1 - g_i)$ for at least one choice of i. Therefore, $\mathbf f (x_1,\ldots ,x_n)$ approximates a product gate (up to a normalizing constant).

1.2 A.2 Proof that $2^n$ Neurons are Necessary

Suppose that S is a subset of $\{1,\ldots ,n\}$ and consider taking the partial derivatives of (A2) and (A3), respectively, with respect to all the variables $\{x_h\}_{h\in S}$. Then, we obtain the equalities

$$\begin{aligned} \frac{n!\, \sigma _n}{(n-|S|)!} \sum _{j=1}^m w_j\prod _{h\in S} a_{hj}\left( \sum _{i=1}^n a_{ij} x_i\right) ^{n-|S|}= & {} \prod _{h\not \in S} x_h, \end{aligned}$$

(A5)

$$\begin{aligned} \frac{k!\, \sigma _k}{(k-|S|)!} \sum _{j=1}^m w_j\prod _{h\in S} a_{hj}\left( \sum _{i=1}^n a_{ij} x_i\right) ^{k-|S|}= & {} 0, \end{aligned}$$

(A6)

for all $0\le k \le n-1$. Let $\mathbf A$ denote the $2^n \times m$ matrix with elements

$$\begin{aligned} A_{Sj} \equiv \prod _{h \in S} a_{hj}. \end{aligned}$$

(A7)

We will show that $\mathbf A$ has full row rank. Suppose, towards contradiction, that $\mathbf c ^t\mathbf A =\mathbf{0}$ for some non-zero vector $\mathbf c$. Specifically, suppose that there is a linear dependence between rows of $\mathbf A$ given by

$$\begin{aligned} \sum _{\ell =1}^r c_{\ell } A_{S_\ell ,j} = 0, \end{aligned}$$

(A8)

where the $S_{\ell }$ are distinct and $c_{\ell } \ne 0$ for every $\ell$. Let s be the maximal cardinality of any $S_{\ell }$. Defining the vector $\mathbf d$ whose components are

$$\begin{aligned} d_j\equiv w_j\left( \sum _{i=1}^n a_{ij} x_i\right) ^{n-s}, \end{aligned}$$

(A9)

taking the dot product of Eq. (A8) with $\mathbf d$ gives

$$\begin{aligned} 0= & {} \mathbf c ^t\mathbf A \mathbf d = \sum _{\ell =1}^r c_{\ell } \sum _{j=1}^m w_j\prod _{h \in S_{\ell }} a_{hj} \left( \sum _{i=1}^n a_{ij} x_i\right) ^{n-s} \nonumber \\= & {} \sum _{\ell \mid (|S_{\ell }| = s)} c_{\ell } \sum _{j=1}^m w_j\prod _{h\in S_{\ell }} a_{hj} \left( \sum _{i=1}^n a_{ij} x_i\right) ^{n-|S_{\ell }|} \\&+ \sum _{\ell \mid (|S_{\ell }| < s)} c_{\ell } \sum _{j=1}^m w_j\prod _{h\in S_{\ell }} a_{hj} \left( \sum _{i=1}^n a_{ij} x_i\right) ^{(n+|S_{\ell }|-s)-|S_{\ell }|}.\nonumber \end{aligned}$$

(A10)

Applying Eq. (A6) (with $k=n+|S_{\ell }|-s$) shows that the second term vanishes. Substituting Eq. (A5) now simplifies Eq. (A10) to

$$\begin{aligned} 0 = \sum _{\ell \mid (|S_{\ell }| = s)} \frac{c_{\ell }(n-|S_\ell |)!}{n!\> \sigma _n} \> \prod _{h\not \in S_{\ell }} x_h, \end{aligned}$$

(A11)

i.e., to a statement that a set of monomials are linearly dependent. Since all distinct monomials are in fact linearly independent, this is a contradiction of our assumption that the $S_{\ell }$ are distinct and $c_{\ell }$ are nonzero. We conclude that $\mathbf A$ has full row rank, and therefore that $m \ge 2^n$, which concludes the proof.