Exactly Predictable Functions for Simple Neural Networks

Abstract

We examine the degree of accuracy of simple feedforward neural nets with N inputs and a single output to forecast time series that represent analytical functions. We show that the subspace of those functions, whose higher order derivatives can be clustered into a finite number of linearly dependent groups, can be forecasted exactly by a neural net. Furthermore, we derive generally applicable summation and product rules that permit us to calculate the associated optimum connection weights for the particular network architecture for complicated but exactly predictable functions. If a general network is initialized with these particular weights, the learning process for general data (with noise) can be significantly accelerated and the forecasting accuracy increased. We also show that neural nets can be used to predict the finite value of diverging sums, which is a generic problem for most perturbation-based approaches to physical systems.

Appendices

Appendix A: Proof of the Superposition Laws for Sums

Here we briefly outline the basic ideas for the general proof of the superposition law that permits us to construct the perfect connection weights $W_{k}(F)$ for $F(t)\equiv f(t) + g(t)$ in terms of the original weights of f(t), denoted by $W_{n}(f)$, and of g(t), denoted by $W_{m}(f)$. We require

$$\begin{aligned} f(t)+g(t)=\; & {} \sum _{k=1}^{N+M}W_{k}(f+g)\left[ f(t-kh)+g(t-kh)\right] \end{aligned}$$

(10)

$$\begin{aligned} f(t)=\; & {} \sum _{n=1}^{N}W_{n}(f) \ f(t-nh) \end{aligned}$$

(11)

$$\begin{aligned} g(t)=\; & {} \sum _{m=1}^{M}W_{m}(g) \ g(t-mh). \end{aligned}$$

(12)

The proof in full generality is very clumsy and can be best performed with some computer algebra software packages such as Mathematica or MatLab. The basic idea is to replace the $(M+1)$ functions at the times f(t), $f(t-h)$ up to $f(t - M h)$ in terms of the N functions at earlier times $f(t-j h)$ with $j = 1+M, N+M$. This iterative procedure that needs to be done in a strict consecutive order is extremely cumbersome. For example, by evaluating both sides of Eq. (11) for the argument $t-M h$, we have

$$\begin{aligned} f(t-Mh)=\sum _{n=1}^{N}W_{n}(f) \ f\left( t-(M+n)h\right) . \end{aligned}$$

(13)

Similarly, for the argument $t-(M-1)h$ and insertion of Eq. (13) we obtain

$$\begin{aligned}&f(t-(M-1)h) =\sum _{n=1}^{N}W_{n}(f) \ f\left( t-(M-1+n)h\right) \nonumber \\&\quad =W_{1}(f) \ f(t-Mh) + \sum _{n=2}^{N}W_{n}(f) \ f\left( t-(M-1+n)h\right) \nonumber \\&\quad =W_{1}(f) \sum _{n=1}^{N}W_{n}(f) \ f\left( t-(M+n)h\right) \nonumber \\&\quad + \sum _{n=2}^{N}W_{n}(f) \ f\left( t-(M-1+n)h\right) . \end{aligned}$$

(14)

This sequence of iterative steps needs to be repeated $(M+1)$ times until the function f(t) can be expressed in terms of $f(t-j h)$ with $j = M+1, N+M$ and all $W_{n}(f)$. The same replacements need to be performed for the function g(t) as well. Here the $(N+1)$ functions at the times g(t), $g(t-h)$ up to $g(t-N h)$ need to be expressed in terms of the N functions at earlier times $g(t-j h)$ with $j=N+1, N+M$.

After these expressions are inserted into Eq. (10), this equation becomes finally a single linear equation for the unknown $(N+M)$ weights $W_{k}(f+g)$, containing all N weights $W_{n}(f)$ and M weights $W_{m}(f)$ as well as the N functions $f(t-j h)$ with $j = 1+M, N+M$ and M functions $g(t-jh)$ with $j = N+1, N+M$.

As this single equation needs to be satisfied for all times t, the $(N+M)$ pre-factors in front of all $f(t-j h)$ and $g(t-j h)$ need to vanish identically. The corresponding set of $(N+M)$ equations for the $(N+M)$ weights $W_{k}(f+g)$ can be solved uniquely.

To give the reader, a better idea of the complexity of the derivation, we present here a concrete example, where we choose $N = 3$ and $M = 2$. For example, $f(t) = t^{2} \exp (3 t)$ and $g(t) = \cos (5t)$ would fall in this category with the known weights according to Table 1.

$$\begin{aligned} f(t)+g(t)=\; & {} \sum _{k=1}^{5}W_{k}\left[ f(t-kh)+g(t-kh)\right] \end{aligned}$$

(15)

$$\begin{aligned} f(t)=\; & {} U_{1} \ f(t-h)+U_{2} f(t-2h)+U_{3} \ f(t-3h) \end{aligned}$$

(16)

$$\begin{aligned} g(t)=\; & {} V_{1} \ g(t-h)+V_{2} \ g(t-2h), \end{aligned}$$

(17)

where for notational simplicity we abbreviate $U_{n}\equiv W_{n}(f)$ and $V_{m}\equiv W_{m}(g)$. Using Eqs. (16) and  (17) repeatedly, the sequence of required replacements leads to

$$\begin{aligned} f(t)=\; & {} \left[ U_{1}^{3}+2U_{1}U_{2}+U_{3}\right] f(t-3h)\nonumber \\&+\left[ U_{2}\left( U_{1}^{2}+U_{2}\right) +U_{1} U_{3}\right] f(t-4h) \nonumber \\+ & {} \left( U_{1}^{2}+U_{2}\right) U_{3}f(t-5h) \nonumber \\ f(t-h)=\; & {} \left[ U_{1}^{2}+U_{2}\right] f(t-3h)+\left( U_{1}U_{2}\right. \nonumber \\&\left. +U_{3}\right) f(t-4h)+U_{1}U_{3}f(t-5h) \nonumber \\ f(t-2h)=\; & {} U_{1}f(t-3h)+U_{2}f(t-4h)+U_{3}f(t-5h) \nonumber \\ g(t)=\; & {} \left( V_{1}^{4}+3V_{1}^{2}V_{2}+V_{2}^{2}\right) \ g(t-4h)\nonumber \\&+V_{1}V_{2}\left( V_{1}^{2}+2V_{2}\right) \ g(t-5h) \nonumber \\ g(t-h)=\; & {} \left( V_{1}^{3}V_{2}+2V_{1}V_{2}\right) \ g(t-4h)\nonumber \\&+V_{2}\left( V_{1}^{2}+V_{2}\right) \ g(t-5h) \nonumber \\ g(t-2h)=\; & {} \left( V_{1}^{2}+V_{2}\right) \ g(t-4h)+V_{1}V_{2}\ g(t-5h) \nonumber \\ g(t-3h)=\; & {} V_{1}\ g(t-4h)+V_{2}\ g(t-5h). \end{aligned}$$

(18)

As a result, we obtain for Eq. (15)

$$\begin{aligned} 0=\; & {} -f(t)-g(t)+ \sum _{k=1}^{5} W_{k}\left[ f(t-k h) + g(t-k h\right] \nonumber \\= \;& {} A_{1}f(t-3h)+A_{2}f(t-4h)\nonumber \\&+A_{3}f(t-5h)+A_{4}\ g(t-4h)+A_{5}\ g(t-5h), \end{aligned}$$

(19)

where the five coefficients are given by

$$\begin{aligned} A_{1}=\; & {} -U_{1}^{3}-U_{3}-2U_{1}U_{2}+\left( U_{1}^{2}+U_{2}\right) W_{1}+U_{1}W_{2}+W_{3} \nonumber \\ A_{2}=\; & {} -U_{2}^{2}-U_{1}\left( U_{1}U_{2}+U_{3}\right) \nonumber \\&+\left( U_{1}U_{2}+U_{3}\right) W_{1}+U_{2}W_{2}+W_{4} \nonumber \\ A_{3}=\; & {} -U_{2}U_{3}-U_{1}U_{1}U_{3}+U_{3}U_{1}W_{1}+U_{3}W_{2}+W_{5} \nonumber \\ A_{4}=\; & {} -V_{1}^{4}-3V_{1}^{2}V_{2}-V_{2}^{2}+(V_{1}^{3}\nonumber \\&+2V_{1}V_{2})W_{1}+\left( V_{1}^{2}+V_{2}\right) W_{2}+V_{1}W_{3}+W_{4} \nonumber \\ A_{5}=\; & {} -V_{1}^{3}V_{2}-2V_{1}V_{2}^{2}+\left( V_{1}^{2}V_{2}\right. \nonumber \\&\left. +V_{2}^{2}\right) W_{1}+V_{1}V_{2}W_{2}+V_{2}W_{3}+W_{5}. \end{aligned}$$

(20)

If we equate these five coefficients $A_{k}$ to zero, we obtain the final solutions for the weights W as

$$\begin{aligned} W_{1}= \;& {} U_{1}+V_{1} \nonumber \\ W_{2}= \;& {} U_{2}+V_{2}-U_{1}V_{1} \nonumber \\ W_{3}=\; & {} U_{3}-U_{1}V_{2}-U_{2}V_{1} \nonumber \\ W_{4}=\; & {} -U_{2}V_{2}-U_{3}V_{1} \nonumber \\ W_{5}=\; & {} -U_{3}V_{2} \end{aligned}$$

(21)

In view of the complexity of the expressions in the intermediate steps, these forms are remarkably simple and they are in full agreement with the general solutions of Eq. (7) for arbitrary N and M.

Appendix B: Weight Factors for the Product Rule

Here we briefly outline the basic ideas for the general proof of the superposition law that permits us to construct the perfect connection weights $W_{k}(F)$ for products $F(t)\equiv f(t)g(t)$ in terms of the original weights of f(t), denoted by $W_{n}(f)$, and of g(t), denoted by $W_{m}(f)$. We require

$$\begin{aligned} f(t)\ g(t)=\; & {} \sum _{k=1}^{NM}W_{k}(fg) f(t-kh)\ g(t-kh) \end{aligned}$$

(22)

$$\begin{aligned} f(t)=\; & {} \sum _{n=1}^{N}W_{n}(f) \ f(t-nh) \end{aligned}$$

(23)

$$\begin{aligned} g(t)=\; & {} \sum _{m=1}^{M}W_{m}(g) \ f(t-mh). \end{aligned}$$

(24)

The approach to derive of the weights $W_{k}(fg)$ from the $W_{k}(f)$ and $W_{k}(g)$ is—in principle—similar to the one used in appendix A, but it is significantly more complicated and we illustrate it here only for the $N=3$ and $M=3$ case. Here we would iteratively use Eq. (23) to replace the functions f and g at later times in terms of the six values $f(t-kh)$ and $g(t-kh)$ for $k = 7, 8$ and 9. After the replacements, the central equation Eq. (23) for the nine weights $W_{k}(fg)$ depend on the nine time-dependent product functions $f(t-k_{1}h)g(t-k_{2}h)$ with $k_{1} = 7, 8, 9$ and $k_{2} = 7, 8, 9$. If we assume that these nine functions are linearly independent of each other, we have to require that the corresponding nine pre-factors vanish. If we solve the resulting nine coupled but linear equations of the nine weights $W_{k}(fg)$ for $k=1, 2, \ldots , 9$ we finally obtain the solutions

$$\begin{aligned} W_{1}(fg)=\; & {} U_{1}V_{1} \end{aligned}$$

(25)

$$\begin{aligned} W_{2}(fg)=\; & {} U_{1}^{2}V_{2}+U_{2}V_{1}^{2}+2U_{2}V_{2} \end{aligned}$$

(26)

$$\begin{aligned} W_{3}(fg)=\; & {} U_{3}\left( V_{1}^{3}+3V_{1}V_{2}+3V_{3}\right) \nonumber \\&+U_{1}\left( U_{2}V_{1}V_{2}+U_{1}^{2}V_{3}+3U_{2}V_{3}\right) \end{aligned}$$

(27)

$$\begin{aligned} W_{4}(fg)=\; & {} U_{1}^{2}U_{2}V_{1}V_{3}-U_{2}^{2}\left( V_{2}^{2}-2V_{1}V_{3}\right) \nonumber \\&+U_{1}U_{3}\left[ V_{2}\left( V_{1}^{2}+2V_{2}\right) +V_{1}V_{3}\right] \end{aligned}$$

(28)

$$\begin{aligned} W_{5}(fg)=\; & {} -U_{1}U_{2}^{2}V_{2}V_{3}+U_{1}^{2}U_{3}\left( V_{1}^{2}\right. \nonumber \\&\left. +2V_{2}\right) V_{3}+U_{2}U_{3}\left( -V_{1}V_{2}^{2}+2V_{1}^{2}V_{3}-V_{2}V_{3}\right) \end{aligned}$$

(29)

$$\begin{aligned} W_{6}(fg)=\; & {} U_{2}^{3}V_{3}^{2}-U_{1}U_{2}U_{3}V_{3}\left( V_{1}V_{2}+3V_{3}\right) \nonumber \\&+U_{3}^{2}\left( V_{2}^{3}-3V_{1}V_{2}V_{3}-3V_{3}^{2}\right) \end{aligned}$$

(30)

$$\begin{aligned} W_{7}(fg)=\; & {} U_{3}V_{3}\left[ U_{2}^{2}V_{1}V_{3}+U_{1}U_{3}\left( V_{2}^{2}-2V_{1}V_{3}\right) \right] \end{aligned}$$

(31)

$$\begin{aligned} W_{8}(fg)=\; & {} -U_{2}U_{3}^{2}V_{2}V_{3}^{2} \end{aligned}$$

(32)

$$\begin{aligned} W_{9}(fg)=\; & {} U_{3}^{3}V_{3}^{3}. \end{aligned}$$

(33)

For notational simplicity, we have used again the abbreviations $U_{k}\equiv W_{k}(f)$ and $V_{k}\equiv W_{k}(g)$. Unfortunately, we have not been able to recognize a certain regular pattern to these 9 weights that would have allowed us to predict the corresponding 16 weights for the $N=4$ $M=4$ system. Even though we note that the sum of the indices of each factor U and V matches the index k of $W_{k}(fg)$, respectively, to predict reliably the corresponding permutations of these factors, their pre-factors and signs seems difficult.

Appendix C: Optimum Weights Involving a Sum of Products

Here we examine the optimum weights due for the specific function f(t)

$$\begin{aligned} f(t)=3\ \exp (a_{1}t)\ \cos (b_{1}t)+\exp (a_{2}t)\cos (b_{2}t), \end{aligned}$$

(34)

where we have used the specific values $a_{1}=-2$ and $a_{2}=2$ for the decay and growth rates and $b_{1} = 70$ and $b_{2} =20$ for the two frequencies. As a sum of the class = 2 functions $\exp (a_{1}t) \cos (b_{1}t)$ and $\exp (-a_{2}t) \cos (b_{2}t)$, the function f(t) is again a epf of class = 4. Applying consecutively the superposition laws derived in this work for optimal weights for the summation and products of functions (Eqs. 7, 25 and  26), one can derive the following analytical expressions for the six optimal weights.

$$\begin{aligned} W_{1}=\; & {} 2\exp (a_{1}h)\cos (b_{1}h)\nonumber \\&+2\exp (a_{2}h)\cos (b_{2}h) \end{aligned}$$

(35)

$$\begin{aligned} W_{2}=\; & {} -\exp (2a_{1}h)-\exp (2a_{2}h)\nonumber \\&-4\exp (a_{1}h+a_{2}h)\cos (b_{1}h)\cos (b_{2}h) \end{aligned}$$

(36)

$$\begin{aligned} W_{3}=\; & {} 2\exp (a_{1}h+2a_{2}h)\cos (b_{1}h)\nonumber \\&+2\exp (2a_{1}h+a_{2}h)\cos (b_{2}h) \end{aligned}$$

(37)

$$\begin{aligned} W_{4}=\; & {} -\exp (2a_{1}h+2a_{2}h). \end{aligned}$$

(38)

Fig. 6

The dependence of the optimal four weights for the test function Eq. (34) for $a_{1}= -2$, $a_{2} = 2$, $b_{1} = 70$ and $b_{2} = 20$ as a function of the grid spacing h

In Fig. 6 we have graphed these four optimum weights as a function of the grid spacing h. In the limit of small spacings $h\rightarrow 0$ we find that the weights approach the values $(W_{1},W_{2},W_{3},W_{4}) =(4, -6, 4, -1)\equiv {\mathbf {W}}(h=0)$. This set corresponds precisely the optimal (h-independent) weights $B_{n4}$ for any polynomial of degree 3. This is not a coincidence as the original optimal weights of the constituent functions $\exp (at)$, $\cos (bt)$ of f(t) of Eq. (34) converge already in this limit to the alternating binomial coefficients given in Eq. (8).

The key question is, whether the binomial coefficients can act as helpful initial values for the learning algorithm for the relevant case where $h\ne 0$. For example, for $h<0.0076$, each the optimal set of weights differs from the set ${\mathbf {W}}(h=0)$ by at most $10\%$. For example, for $h=0.01$ we have the set of exact optimal weights given by ${\mathbf {W}}(0.01) =(3.50, -5.00, 3.48, -1.00)$, very similar to $B_{n4}=(-1)^{n+1}(4,n)$. This suggests, that as long as the grid spacing is not too large, the binomial set should be an ideal set of weight parameters to initialize the net for the learning process.