Approximation and estimation bounds for artificial neural networks

Abstract

For a common class of artificial neural networks, the mean integrated squared error between the estimated network and a target functionf is shown to be bounded by

$$O\left( {\frac{{\mathop c\nolimits_f^2 }}{n}} \right) + O\left( {\frac{{nd}}{N}\log N} \right)$$

wheren is the number of nodes,d is the input dimension of the function,N is the number of training observations, andC _f is the first absolute moment of the Fourier magnitude distribution off. The two contributions to this total risk are the approximation error and the estimation error. Approximation error refers to the distance between the target function and the closest neural network function of a given architecture and estimation error refers to the distance between this ideal network function and an estimated network function. Withn ∼ C _f (N/(d logN))^1/2 nodes, the order of the bound on the mean integrated squared error is optimized to beO(C _f ((d/N) logN)^1/2). The bound demonstrates surprisingly favorable properties of network estimation compared to traditional series and nonparametric curve estimation techniques in the case thatd is moderately large. Similar bounds are obtained when the number of nodesn is not preselected as a function ofC _f (which is generally not knowna priori), but rather the number of nodes is optimized from the observed data by the use of a complexity regularization or minimum description length criterion. The analysis involves Fourier techniques for the approximation error, metric entropy considerations for the estimation error, and a calculation of the index of resolvability of minimum complexity estimation of the family of networks.