REVIEW Communicated by Yann LeCun

First- and Second-Order Methods for Learning:

Between Steepest Descent and Newtons Method

Roberto Battiti
Dipartimento di Matematica, Universita di Trento,
38050 Povo (Trento),Italy

On-line first-order backpropagation is sufficiently fast and effective for

many large-scale classification problems but for very high precision
mappings, batch processing may be the method of choice. This pa-
per reviews first- and second-order optimization methods for learning
in feedfomard neural networks. The viewpoint is that of optimiza-
tion: many methods can be cast in the language of optimization tech-
niques, allowing the transfer to neural nets of detailed results about
computational complexity and safety procedures to ensure convergence
and to avoid numerical problems. The review is not intended to de-
liver detailed prescriptions for the most appropriate methods in specific
applications, but to illustrate the main characteristics of the different
methods and their mutual relations.

1 Introduction

There are cases in which learning speed is a limiting factor in the prac-
tical application of multilayer perceptrons to problems that require high
accuracy in the network mapping function. In this class are applications
related to system identification and nonlinear modeling, time-series pre-
diction, navigation, manipulation, and robotics. In addition, the stan-
dard batch backpropagation (BP) method ( e g , Rumelhart and McClel-
land 1986) requires a selection of appropriate parameters by the user,
that is mainly executed with a trial-and-error process. Since one of the
competitive advantages of neural networks is the ease with which they
may be applied to novel or poorly understood problems, it is essential to
consider automated and robust learning methods with a good average
performance on many classes of problems.
This review describes some methods that have been shown to acceler-
ate the convergence of the learning phase on a variety of problems, and
suggests a possible taxonomy of the different techniques based on their
order (i.e., their use of first or second derivatives), space and computa-
tional requirements, and convergence properties. Some of these methods,
while requiring only limited modifications of the standard BP algorithm,

Neural Computation 4,141-166 (1992) @ 1992 Massachusetts Institute of Technology

142 Roberto Battiti

yield a speed-up of very large factors and, furthermore, are easier to

apply because they do not require the choice of critical parameters (like
the learning rate) by the neural network practitioners.
The presentation attempts a classification of methods derived from
the literature based on their underlying theoretical frameworks, with par-
ticular emphasis on techniques that are appropriate for the supervised
learning of multilayer perceptrons (MLPs).
The general strategy for the supervised learning of an input-output
mapping is based on combining a quickly convergent local method with
a globally convergent one. Local methods are based on appropriate local
models of the function to be minimized. In the following sections, first
we consider the properties of methods based on a linear model (steepest
descent and variations), then we consider methods based on a quadratic
model (Newtons method and approximations).

2 Backpropagation and Steepest Descent

The problem of learning an input-output mapping from a set of P ex-

amples can be transformed into the minimization of a suitably defined
error function. Although different definitions of the error have been used,
for concreteness we consider the traditional sum-of-squared-differences
error function defined as

(2.1)

where t,, and oprare the target and the current output values for pattern
p , respectively, and no is the number of output units. The learning pro-
cedure known as backpropagation (Rumelhart and McClelland 1986) is
composed of two stages. In the first, the contributions to the gradient
coming from the different patterns (dEp/8wij) are calculated backpropa-
gating the error signal. The partial contributions are then used to correct
the weights, either after every pattern presentation (on-line BP), or after
they are summed in order to obtain the total gradient (batch BP).
Let us define as gk the gradient of the error function kk = VE(wk)].
The batch backpropagation update is a form of gradient descent defined
as

while the on-line update is

It is not unusual to observe speed-ups of 100-1000 with respect to BP with fixed

learning and momentum rates.
First- and Second-Order Methods for Learning 143

If the learning rate E tends to zero, the difference between the weight
vectors Wk+,,during one epoch of the on-line method tends to be small
and the step EVE~(W~+,,) induced by a particular pattern p can be ap-
proximated by cVEp(wk)(by calculating the gradient at the initial weight
vector). Summing the contributions for all patterns, the movement in
weight space during one epoch will be similar to the one obtained with a
single batch update. However, in general the learning rate has to be large
to accelerate convergence, so that the paths in weight space of the two
methods differ.
The on-line procedure has to be used if the patterns are not available
before learning starts [see, for example, the perceptron used for adaptive
equalization in Widrow and Stearns (1985)1, and a continuous adapta-
tion to a stream of input-output signals is desired. On the contrary, if
all patterns are available, collecting the total gradient information before
deciding the next step can be useful in order to avoid a mutual interfer-
ence of the weight changes (caused by the different patterns) that could
occur for large learning rates (this effect is equivalent to a sort of noise in
the true gradient direction). One of the reasons in favor of the on-line
approach is that it possesses some randomness that may help in escaping
from a local minimum. The objection to this is that the method may, for
the same reason, miss a good local minimum, while there is the alter-
native method of converging to the "nearest" local minimum and using
randomness2 to escape only after convergence. In addition, the on-line
update may be useful when the number of patterns is so large that the er-
rors involved in the finite precision computation of the total gradient may
be comparable with the gradient itself. This effect is particularly present
for analog implementations of backpropagation, but it can be controlled
in digital implementations by increasing the number of bits during the
gradient accumulation. The fact that many patterns possess redundant
information [see, for example, the case of hand-written character recog-
nition in LeCun (1986)l has been cited as an argument in favor of on-line
BP, because many of the contributions to the gradient are similar, so that
waiting for all contributions before updating can be wasteful. In other
words, averaging over more examples to obtain a better estimate of the
gradient does not improve the learning speed sufficiently to compensate
for the additional computationaI cost of taking these patterns into ac-
count. Nonetheless, the redundancy can also be limited using batch BP,
provided that learning is started with a subset of "relevant" patterns and
continued after convergence by progressively increasing the example set.
This method has for example been used in Kramer and Sangiovanni-
Vicentelli (1988) for the digit recognition p r ~ b l e m .Even
~ if the training

21n addition, there are good reasons why random noise may not be the best available
alternative to escape from a local minimum and avoid returning to it. See, for example,
the recently introduced TABU methods (Glover 1987).
31f the redundancy is clear (when for example many copies of the sume example are
present) one may preprocess the example set in order to eliminate the duplication. On
144 Roberto Battiti

set is redundant, on-line BP may be slow relative to second-order meth-

ods for badly conditioned problems. The convergence of methods based
on gradient descent (or approximations thereof) depends critically on the
relative size of the maximum and minimum eigenvalues of the Hessian
matrix [see LeCun et al. (1991) and equation 2.6 for the case of steep-
est descent]. This is related to the "narrow valley" effect described in
Rumelhart and McClelland (1986). In addition, the batch approach lends
itself to straightforward modifications using second-order information,
as it will be shown in the following sections.
At this point, in order not to mislead the reader in the choice of the
most appropriate method for a specific application, it is useful to remem-
ber that many large-scale experiments (mainly in pattern recognition-
classification) have used the simple on-line version of BP with full satis-
faction, considering both the final result and the number of iterations. In
some cases, with a careful tuning of the on-line procedure the solution
is reached in a very small number of epochs, that is, in a few presenta-
tions of the complete example set [see, for example, Rumelhart and Mc-
Clelland (198611 and it is difficult to reach a comparable learning speed
with batch techniques (Cardon ef al. 1991). Tuning operations are, for
example, the choice of appropriate parameters like the learning and mo-
mentum rate (Fahlman 1988), "annealing" schedules for the learning rate
(that is progressively reduced) (Malferrari et al. 19901, updating schemes
based on summing the contributions of related patterns (Sejnowski and
Rosenberg 1986), "small batches," "selective" corrections only if the error
is larger than a threshold (that may be progressively reduced) (Vincent
1991; Allred and Kelly 19901, randomization of the sequence of pattern
presentation, etc. The given references are only some examples of signif-
icant applications, out of an abundant literat~re.~ Because in these cases
only approximated output values are required and the example set often
is characterized by a large degree of redundancy, these two attributes
could be considered as votes in favor of on-line BP, again provided that
the trial-and-error phase is not too expensive.
In the original formulation, the learning rate 6 was taken as a fixed
parameter. Unfortunately, if the learning rate is fixed in an arbitrary way,
there is no guarantee that the net will converge to a point with vanishing
gradient. Nonetheless, convergence in the on-line approach can be ob-
tained by appropriately choosing a fixed and sufficiently small learning
rate.
The issue of an appropriate fixed learning rate for on-line LMS learn-
ing has been investigated in the adaptive signal processing literature

the contrary, if redundancy is only partial, the redundant patterns have to be presented
to and learned by the network in both versions.
4The availability of many variations of the on-line technique is one of the reasons
why "fafi" comparisons with the batch and second-order versions are complex. Which
version has to be chosen for the comparison? If the final convergence results have been
obtained after a tuning process, should the tuning times be included in the comparison?
First- and Second-Order Methods for Learning 145

[see, for example, Bingham (1988)l. The result is that the convergence of
stochastic LMS is guaranteed if E < 1 / ( N G), where N is the number
of parameters being optimized and A, is the largest eigenvalue of the
autocorrelation function of the input.5 A detailed study of adaptive filters
is presented in Widrow and Stearns (1985). The effects of the autocor-
relation matrix of the inputs on the learning process (for a single linear
unit) are discussed in LeCun et al. (1991). In this framework the appro-
priate learning rate for gradient descent is 1/Amax. These results cannot
be extended to multilayer networks (with nonlinear transfer functions)
in a straightforward manner, but they can be used as a starting point for
useful heuristics.
The convergence properties of the LMS algorithm with adaptive learn-
ing rate are presented in Luo (1991), together with a clear comparison of
the LMS algorithm with stochastic gradient descent and adaptive filter-
ing algorithms. The main result is that if the learning rate E , for the nth
training cycle satisfies the two conditions:

(2.4)
n=l n=l

then the sequence of weight matrices generated by the LMS algorithm

(with a cyclic pattern presentation) will converge to the optimal solution
(minimizing the mean-square error).
But even if E is appropriately chosen so that the error decreases with
a reasonable speed and oscillations are avoided, gradient descent is not
always the fastest method to employ. This is not an intuitive result,
because the negative gradient is the direction of fastest decrease in the
error. Unfortunately, the greed in trying to reach the minimum along
this one-dimensional direction is paid at the price that subsequent gradi-
ent directions tend to interfere, so that in a weight space with dimension
N, the one-dimensional minimization process has to be repeated for a
number of times that is normally much higher than N (even for sim-
ple quadratic functions).6 In the steepest descent method, the process of
minimizing along successive negative gradients is described as:

where f k minimizes E ( w ~- E g k ) . If steepest descent is used to minimize

+
a quadratic function Q(w) = cTw ;wTGw (G symmetric and positive

5l owe this remark to the referee.

61t is easy to show that, if exact one-dimensional optimization along the negative
gradient is used, the gradient at the next step is perpendicular to the previous one.
If, considering an example in two dimensions, the lines at equal error are given by
elongated ellipsoids, the system, for a general starting point, goes from a point to the
one that is tangent to the equal-error lines along the gradient, and then repeats along a
perpendicular direction.
146 Roberto Battiti

definite), it can be shown that

IQ(wk+i) - Q(w*)l M ( 7)max

Vmax
- rlmin
+ Vmin ) 2 lQ(wk) - Q(w*)l (2.6)

where rimax and qmln are the maximum and minimum eigenvalues of the
matrix G, and wl; is the minimizer [see Luenberger (197311. If these two
eigenvalues are very different, the distance from the minimum ~ a l u e
is multiplied each time by a number that is close to one. The type of
convergence in equation 2.6 is termed q-linear convergence. One has 9-
superlinear convergence if, for some sequence ck that converges to 0, one
has

IQ(W+l) - Q(w*)I 5 C k I Q ( W ) - Q(w*)I (2.7)

Finally, wk is said to converge with q-order at least p if, for some constant
c, one has

I Q ( w +I ) - Q(w*)l 5 clQ(wk) - Q(w*)lP (2.8)

In practice q-linear convergence tends to be very slow, while q-superlinear
or q-quadratic ( p = 2) convergence is eventually much faster7
In the following sections, we illustrate some techniques that can be
used to ensure convergence, to avoid numerical problems related to
finite-precision computation, and to accelerate the minimization process
with respect to standard batch BP.

3 Conjugate Gradient Methods

Let us consider a quadratic function Q ( w ) of the type described in the

previous section. We have seen that one of the difficulties in using the
steepest descent method is that a one-dimensional minimization in di-
rection a followed by a minimization in direction b does nut imply that
the function is minimized on the subspace generated by a and b. Mini-
mization along direction b may in general spoil a previous minimization
along direction a (this is why the one-dimensional minimization in gen-
eral has to be repeated a number of times much larger than the number
of variables). On the contrary, if the directions were noninterfering and
linearly independent, at the end of N steps the process would converge
to the minimum of the quadratic function.
The concept of noninterfering directions is at the basis of the conjugate
gradient method (CG) for minimization. Two directions are mutually
conjugate with respect to the matrix G if
pTGpj = O when i#J (3.1)
7For example, for a q-quadratically convergent method, the number of significant
digits in the solution is doubled after each iteration.
First- and Second-Order Methods for Learning 147

After minimizing in direction pi, the gradient at the minimizer will be

perpendicular to p i . If a second minimization is in direction p i + l , the
change of the gradient along this direction is gi+l -gi = aGp,+l (for some
constant a). The matrix G is indeed the Hessian, the matrix containing
the second derivatives, and in the quadratic case the model coincides
with the original function. Now, if equation 3.1 is valid, this change is
perpendicular to the previous direction [pT(gitl - gi) = 01, therefore the
gradient at the new point remains perpendicular to pi and the previous
minimization is not spoiled. While for a quadratic function the conjugate
gradient method is guaranteed to converge to the minimizer in at most
(N + 1) function and gradient evaluations (apart from problems caused
by the finite precision), for a general function it is necessary to iterate the
method until a suitable approximation to the minimizer is obtained.
Let us introduce the vector Y k = &+I - gk. The first search direc-
tion p l is given by the negative gradient -gl. Then the sequence wk of
approximations to the minimizer is defined in the following way:
(3.2)
(3.3)
where g k is the gradient, (Yk is chosen to minimize E along the search
direction p k and /3k is defined by

(3.4)

There are different versions of the above equation. In particular, the

Polak-Ribiere choice is A = ylgk+l the Fletcher-Reeves choice is
They all coincide for a quadratic function (Shanno
p k = g[++1gk+l/gkTgk*
1978). A major difficulty with all the above iorms is that, for a general
function, the obtained directions are not necessarily descent directions
and numerical instability can result.
Although for a wide class of functions the traditional CG method with
exact searches and exact arithmetic is superlinearly convergent, implemen-
tations of the conjugate-gradient method with finite precision compu-
tation are nearly always linearly convergent (Gill et al. 1981), but the
number of steps is in practice much smaller than that required by steepest
descent.
The use of a momentum term to avoid oscillations in Rumelhart and
McClelland (1986) can be considered as an approximated form of con-
jugate gradient. In both cases, the gradient direction is modified with a
term that takes the previous direction into account, the important differ-
ence being that the parameter p in conjugate gradient is automatically
defined by the algorithm, while the momentum rate has to be guessed by
the user. Another difficulty related to the use of a momentum term is due
to the fact that there is an upper bound on the adjustment caused by the
momentum. For example, if all partial derivatives are equal to 1, then the
148 Roberto Battiti

exponentially weighted sum caused by the momentum rate a converges

to 1/(1 - a ) [see Jacobs (1988) for details].' Furthermore, summing the
momentum term to the one proportional to the negative gradient may
produce an ascent direction, so that the error increases after the weight
update.
Among the researchers using conjugate gradient methods for the MLP
are Barnard and Cole (1988), Johansson et al. (19901, Bengio and Moore
(19891, Drago and Ridella (1991), Hinton's group in Toronto, the groups at
CMU, Bell Labs, etc. A version in which the one-dimensional minimiza-
tion is substituted by a scaling of the step that depends on success in error
reduction and goodness of a one-dimensional quadratic approximation
is presented in Moller (1990) (SCG).This O ( N )scheme incorporates ideas
from the "model-trust region" methods (see Section 4.3) and "safety" pro-
cedures that are absent in the CG schemes, yielding convergence results
that are comparable with the OSS method described in Section 6. Some
modifications of the method are presented in Williams (1991). It is worth
stressing that expensive one-dimensional searches are also discouraged
by current results in optimization (see Section 4.1): the search can be
executed using only a couple of function and gradient evaluations.

4 Newton's Method

Newton's method can be considered as the basic local method using

second-order information. It is important to stress that its practical ap-
plicability to multilayer perceptrons is hampered by the fact that it re-
quires a calculation of the Hessian matrix (a complex and expensive
task'). Nonetheless, the method is briefly illustrated because most of
the "useful" second-order methods originate from it as approximations
or variations.
It is based on modeling the function with the first three terms of the
Taylor-series expansion about the current point wc:

and solving for the step sN that brings to a point where the gradient of
the model is zero: Vmc(w, + s N ) = 0. This corresponds to solving the

addition, B. Pearlmutter has recently shown that momentum even if chosen

"optimally" can do no better than q-linear convergence (see his poster at the NIPS 1991
conference).
9A "brute force" method to calculate H is that of using a finite-difference formula.
If the gradient is available (as is the case for feedforward nets), one may use: H , =
[VE(w,+ h e ) - VE(w,)]/h,, with suitable h, steps [see Dennis et al. (1981)l. Note that
N + 1 grad& computations are needed, so that the method is not suggested for large
networks!
First- and Second-OrderMethods for Learning 149

following linear system:

V2E(w,)sN= -VE(w,) (4.2)
sN is, by definition, Newtons step (and direction).
If the Hessian matrix (V2E or H, for short) is positive definite and the
quadratic model is correct, one iteration is sufficient to reach the mini-
mum. Because one iteration consists in solving the linear system in equa-
tion 4.2, the complexity of one step is O(N3),using standard methods.O
In general, if the initial point wo is sufficiently close to the minimizer
w,, and V2E(w,)is positive definite, the sequence generated by repeat-
ing Newtons algorithm converges q-quudruticully to w* [see Dennis and
Schnabel (1983) for details].
Assuming that the Hessian matrix can be obtained in reasonable com-
puting times, the main practical difficulties in applying the pure New-
tons method of equation 4.2 arise when the Hessian is not positive defi-
nite, or when it is singular or ill-conditioned. If the Hessian is not positive
definite (this may be the case in multilayer perceptron learning!), there
is no natural scaling in the problem: there are directions p k of negative
curvature (i.e., such that $Hpk 5 0) that would suggest infinite steps
in order to minimize the model. Unfortunately, long steps increase the
probability of leaving the region where the model is appropriate, produc-
ing nonsense. This behavior is not uncommon for multilayer perceptron
learning: in some cases a local minimization step increases some weights
by large amounts, pushing the output of the sigmoidal transfer function
into the saturated region. When this happens, some second derivatives
are very small and, given the finite machine precision or the approxima-
tions, the calculated Hessian will not be positive definite. Even if it is,
the linear system of equation 4.2 may be seriously ill-conditioned. Mod-
ified Newtons methods incorporate techniques for dealing with the above
problems, changing the model Hessian in order to obtain a sufficiently
positive definite and non-singular matrix.
It is worth observing that, although troublesome for the above rea-
sons, the existence of directions of negative curvature may be used to
continue from a saddle point where the gradient is close to zero.
While calculating the analytic gradient for multilayer perceptrons can
be efficiently executed backpropagating the error, calculating the Hes-
sian is computationally complex, so that practical methods have to rely
on suitable approximations. In the following sections we illustrate some
modifications of Newtons method to deal with global convergence, in-
definite Hessian, and iterative approximations to the Hessian itself. In the
review by White (1989) the use of appropriate modifications of Newtons
methods for learning is considered starting from a statistical perspective.
~ ~ )Press
OA smaller upper bound f o r matrix inversion is actually O ( T I O ~see ; et a!.
(1988) for details.
Only second-order methods provide this possibility, while methods based on steep-
est descent are condemned to failure in this case: how many local minima are in
reality saddle points!
150 Roberto Battiti

4.1 Globally Convergent Modifications: Line Searches. Consider-

ing second-order methods, their fast local convergence property suggests
that they are used when the approximation is close to the minimizer. On
the other hand, getting from an initial configuration (that may be very
different from the minimizer) to a point where the local model is accurate
requires some additional effort. The key idea to obtain a general-purpose
successful learning algorithm is tbat of combining a fast tactical local
method with a robust strategic method that assures global convergence.
Because Newton's method (or its modifications when the analytic
Hessian is not available) has to be used near the solution, the suggested
method is that of trying the Newton step first, deciding whether the ob-
tained point is acceptable or not, and backtracking in the last case (i.e.,
selecting a shorter step in the Newton direction).
One of the reasons for searching the next point along the Newton
direction is that this is a descent direction, that is, the value of the error is
guaranteed to decrease for sufficiently small steps along that direction.
It is easy to see why: because the Hessian (and therefore its inverse) is
symmetric and positive definite, the directional derivative of the error is
negative:
dE
-(wc
dX
+ AsN)= V E ( W , ) ~ =S ~- V E ( W , ) ~ H ~ ' V E ( W ,<) 0
If the analytic Hessian has to be approximated, it is essential to consider
only approximations that maintain symmetry and positive definiteness, so
that the one-dimensional minimization step remains unaltered.
When using a line-search algorithm outside the convergence region of
Newton's method, some simple prescriptions have to be satisfied in order
to obtain global convergence. Let the accepted step along the search di-
rection p . be A, the requirement that E(w,+ Acpc):. E(w,) is not sufficient:
the sequence wk may not converge, or it may converge to a point where
the gradient is different from zero. During each one-dimensional search
the steps must decrease the error by a sufficient amount with respect to
the step length (I) and they must be long enough to ensure sufficient
progress (11). Furthermore, the search direction must be kept sufficiently
away from being orthogonal to the gradient.
A formulation of the above requirements that is frequently used in
optimization is based on the work of Armijo and Goldstein [see, for
example, Goldstein (1967)l. Requirement (I) and (11) become
(4.3)
where cy is a fixed constant E ( 0 , l ) and X > 0,

V E ( w , + &p)Tp 2. B V E ( w c ) T p (4.4)
where 0is a fixed constant E ( a ,1). The condition 0 > (1: assures that the
two requirements can be simultaneously satisfied.
First- and Second-Order Methods for Learning 151

If the two above conditions are satisfied at each iteration and if the
error is bounded below, the sequence wk obtained is such that
limk,, VE(wk) = 0, provided that each step is kept away from orthogo-
nality to the gradient (limk,= VE(wk)'sk/llskll # 0).
This result is quite important: we are permitted to use fast approxi-
mated one-dimensional searches without losing global convergence. Re-
cent computational tests show that methods based on fast one-dimen-
sional searches in general require much less computational effort than
methods based on sophisticated one-dimensional minimizations.I2
The line-search method suggested in Dennis and Schnabel (1983) is
well suited for multilayer perceptrons (where the gradient can be ob-
tained with limited effort during the computation of the error) and re-
quires only a couple of error and gradient evaluations per iteration, in
the average. The method is based on quadratic and cubic interpolations
and designed to use in an efficient way the available information about
the function to be minimized [see Dennis and Schnabel(1983) for details].
A similar method based on quadratic interpolation is presented in Battiti
(1989).

4.2 Indefinite Hessians: Modified Cholesky Factorization. When

the Hessian matrix in the local model introduced in equation 4.1 is not
positive definite and well conditioned, equation 4.2 cannot be used with-
out modifications. This can be explained by introducing the spectral
decomposition of the Hessian (based on the availability of eigenvalues
and eigenvectors), writing the matrix H as a sum of projection operators:

(4.5)

where A is diagonal (All is the eigenvalue rli) and U orthonormal. It is

easy to see that, if some eigenvalues are close to zero (with respect to the
largest eigenvalue), the inverse matrix has eigenvalues close to infinity,
a sure source of numerical problem^.'^ If one eigenvalue is negative, the
quadratic model does not have a minimum, because large movements in
the direction of the corresponding eigenvector decrease the error value
to arbitrarily negative values.
A recommended strategy for changing the model Hessian in order
to avoid the above problems is that of summing to it a simple diagonal
matrix of the form pcI (I being the identity matrix), so that [V2E(w,)+ p c I ]
121nsimple words: it does not pay to use a method that requires a limited number
of iterations if each iteration requires a huge amount of computation.
I3ln detail, the conditioning number &(A)of a matrix A is defined as l/All lIA-'JJ,
where 11 11 is the matrix operator norm induced by the vector norm. The condition-
ing number is the ratio of the maximum to the minimum stretch induced by A and
measures, among other effects, the sensitivity of the solution of a linear system to
finite-precision arithmetic, that is of the order of &(A)(machine precision). K ( A )does not
depend on scaling of the matrix by a fixed constant.
152 Roberto Battiti

is positive definite and safely well conditioned. A proper value for /I, can
be efficiently found using the modified Cholesky factorization described
in Gill et al. (1981) and the heuristics described in Dennis and Schnabel
(1983). The resulting algorithm is as follows.
The Cholesky factors of a positive-definite symmetric matrix can be
considered as a sort of "square root" of the matrix. The original matrix
M is expressed as the product LDL', where L is a unit lower triangu-
lar matrixI4 and D is a diagonal matrix with strictly positive diagonal
elements. Taking the square root of the diagonal elements and form-
ing with them the matrix D'12, the original matrix can be written as
M = LD1/2D1/2LT = ITT = R'R, where L is a general lower triangular
matrix, and R a general upper-triangular matrix. The Cholesky factoriza-
tion can be computed in about i N 3 multiplications and additions and is
characterized by good numerical stability. If the original matrix is not pos-
itive definite, the factorization can be modified in order to obtain factors
L and D with all the diagonal elements in D positive and all the elements
in L uniformly bounded. The obtained factorization corresponds to the
factors of a matrix n,differing from the original one only by a diagonal
matrix K with nonnegative elements:
gc= LDLT = H, + K (4.6)
where

A suitable choice of P is described in Gill et al. (1981).15

The availability of the modified Cholesky factorization is the starting
point for the algorithm described in Dennis and Schnabel (1983) to find
pc 2 0 such that V2E(w,) + pcI is positive definite and well conditioned.
Considering the eigenspace of V2E(w,), pC has to be slightly larger than
the magnitude of the most negative eigenvalue. If the matrix E is zero,
pc is set to zero, otherwise it is set to the minimum of two upper bounds.
One upper bound is
u1 = max {ki;}
l<r<N

In fact, the magnitude of the most negative eigenvalue of VE(w,) must be

less than the maximum kfi, because after summing kii to it the eigenvalue
becomes positive (remember that the modified factorization produces a
positive definite matrix). The other upper bound is derived from the
I4A unit lower triangular matrix has the diagonal elements equal to one and all the
elements above the diagonal equal to zero.
15Theprescription is p2 = max {y,t / m ,EM), where y is the largest in magnitude
of the diagonal elements of H,, ( the largest in magnitude of the off-diagonal elements,
and 6 , the
~ machine precision. This result is obtained by requiring that the diago-
nal modification is minimal (minimal norm of K ) , and that sufficiently positive-definite
matrices H, are left unaltered ( K null in this case).
First- and Second-Order Methods for Learning 153

Gerschgorin circle theorem and is defined as

N

where D is a positive factor needed to take the finite precision of compu-

tation into account.16
If u2 is added to the diagonal elements, the matrix becomes strictly
diagonally dominant (hii - C;=l..zilhijl > 0) and therefore positive definite.
The algorithm uses pc = rnin(u1, UZ}.
Software routines for the Cholesky decomposition can be found in
Dennis and Schnabel (1983) or in Dongarra et al. (1979).

4.3 Relations with Model-Trust Region Methods. Up to now we

have considered optimization techniques based on finding a search Airec-
tion and moving by an acceptable amount in that direction ("step-length-
based methods"). In Newton's method the direction was obtained by
"multiplying" the gradient by the inverse Hessian, and the step was the
full Newton step when possible (to obtain fast local convergence) or a
shorter step when the global strategy required it. In Section 4.2 we de-
scribed methods to modify the Hessian when this was not positive def-
inite. Because the modification consisted in adding to the local model a
term quadratic in the Step magnitude (mmodified(WcfS) = m c ( W c f S ) + p c S T s ) ,
one may suspect that minimizing the new model is equivalent to min-
imizing the original one with the constraint that the step s is not too
large. Now, while in line-search algorithms the direction is maintained
and only the step length is changed, there are alternative strategies based
on choosing first a step length and then using the f u l l quadratic model (not
just the one-dimensional one of equation 4.1) to determine the appropri-
ate direction. These competitive methods are called "model-trust-region
methods" with the idea that the model is trusted only within a region,
that is updated using the experience accumulated during the search pro-
cess.
The above suspicion is true and there is indeed a close relationship
between "trust-region" methods and "line-search" methods with diag-
onal modification of the Hessian. This relationship is described by the
following theorem. Suppose that we are looking for the step sc that solves
1
+
min mc(wc+ s) = E(w,) V E ( W , ) ~
+S-sTHcs;
2
subject to IIs(I 5 6, (4.8)
the above problem is solved by
S ( P ) = -(Hc + /LI)-' VE(wC) (4.9)
1 6 is~ 6 (maxa,- minev), CM being the smallest positive number E such that l + c >1
and nruxev, mineu being estimates of the maximum and minimum eigenvalues.
154 Roberto Battiti

for the unique p 1 0 such that the step has the maximum allowed length
(~ S(.L )I = &I, unless the step with p = 0 is inside the trusted region
Clls(0)ll 5 &I, in which case s(0) is the solution, equal to the Newton
step. We omit the proof and the usable techniques for finding b, leaving
the topics as a suggested reading for those in search of elegance and
inspiration [see, for example, Dennis and Schnabel (1983)l.
As a final observation, note that the diagonal modification to the Hes-
sian is a sort of compromise between gradient descent and Newton's
method: When 11 tends to zero the original Hessian is (almost) positive
definite and the step tends to coincide with Newton's step; when p has
to be large the diagonal addition pI tends to dominate and the step tends
to one proportional to the negative gradient:
1
~ ( p=) -(Hc + /LI)-'QE(w,) x --QE(wr)
I"
There is no need to decide from the beginning about whether to use the
gradient as a search direction; the algorithm takes care of selecting the
direction that is appropriate for a local configuration of the error surface.
While not every usable multilayer perceptron needs to have thou-
sands of weights, it is true that this number tends to be large for some
interesting applications. Furthermore, while the analytic gradient is eas-
ily obtained, the calculation of second derivatives is complex and time-
consuming.
For these reasons, the methods described above, while fundamental
from a theoretical standpoint, have to be simplified and approximated in
suitable ways that we describe in the next two sections.

5 Secant Methods

When the Hessian is not available analytically, secant rneth~ds'~are widely

used techniques for approximating the Hessian in an iterative way us-
ing only information about the gradient. In one dimension the second
derivative a2E(w)/L3w2can be approximated with the slope of the secant
line (therefore the term "secant") through the values of the first deriva-
tives in two near points:
@E(W)
(5.1)
dW2

In more dimensions the situation is more complex. Let the current and
next point be wc and w+;defining yc = QE(w+)-QE(w,) and s, = w+ -wc;
the analogous secant equation to equation 5.1 is
H+ s, = yc (5.2)
17Historicallythese methods were called quasi-Nmton methods. Here we follow the
terminology of Dennis and Schnabel (1983), where the term quasi-Newton refers to all
algorithms "derived" from Newton's method.
First- and Second-Order Methods for Learning 155

The new problem is that in more than one dimension equation 5.2 does
not determine a unique H+ but leaves the freedom to choose from a
(V - N)-dimensional affine subspace Q(sc,y c ) of matrices obeying equa-
tion 5.2. The new suggested strategy is that of using equation 5.2 not to
determine but to update a previously available approximation. In particu-
lar, Broyden's update is based on a least change principle: find the matrix
in Q(sc,yc)that is closest to the previously available matrix. This is ob-
tained by projecting18 the matrix onto Q(s,, yc). The resulting Broyden's
update is

(5.3)

Unfortunately, Broyden's update does not guarantee a symmetric matrix.

For this reason, its use in optimization is strongly discouraged (unless
we are willing to live with directions that are not descent directions, a
basic requirement of line-search methods).
Projecting the matrix obtained with Broyden's update onto the sub-
space of symmetric matrices is not enough: the new matrix may be out
of Q(sc,yc).Fortunately, if the two above projections are repeated, the
obtained sequence of matrices (H+),, converges to a matrix that is both
symmetric and in Q(sc,yc).This is the symmetric secant update of Powell:

The Powell update is one step forward, but not the solution. In the
previous sections we have shown the importance of having a symmetric
and positive definite approximation to the Hessian. Now, one can prove
that H+ is symmetric and positive definite if and only if H+ = f+.f:
for some nonsingular matrix f+. Using this fact, one update of this kind
can be derived, expressing H+ = f+ f: and using Broyden's method to
derive a suitable ]+.I9 The resulting update is historically known as the
Broyden, Fletcher, Coldfarb, and Shanno (BFGS) update (by Broyden et
al. 1973) and is given by

(5.5)

The BFGS positive definite secant update has been the most successful
update in a number of studies performed during the years. The positive
definite secant update converges q-superlinearly [a proof can be found

I6The changes and the projections are executed using the Frobenius norm: IlHll~=
(C,,,
h;)''', the matrix is considered as a "long" vector.
I9The solution exists if ycsc > 0, that is guaranteed if "accurate" line searches are
performed (see Section 4.1).
156 Roberto Battiti

in Broyden et al. (1973)l. It is common to take the initial matrix HOas the
identity matrix (first step in the direction of the negative gradient). It is
possible to update directly the Cholesky factors (Goldfarb 1976), with a
total complexity of O ( P ) [see the implementation in Dennis and Schn-
abel (198311. Secant methods for learning in the multilayer perceptron
have been used, for example, in Watrous (1987). The O ( p ) complexity
of BFGS is clearly a problem for very large networks, but the method
can still remain competitive if the number of examples is very large,
so that the computation of the error function dominates. A compari-
son of various nonlinear optimization strategies can be found in Webb
et al. (1988). Second-order methods in continuous time are considered in
Parker (1987).

6 Closing the Gap: Second-Order Methods with 0 0 Complexity -

One drawback of the BFGS update of equation 5.5 is that it requires

storage for a matrix of size N x N and a number of calculations of order
O(N2).20Although the available storage is less of a problem now than it
was a decade ago [for a possible method to cope with limited storage,
see, for example, Nocedal (198011, the computational problem still exists
when N becomes of the order of one hundred or more.
Fortunately, it is possible to kill two birds with one stone. In Battiti
(1989) it is shown that it is possible to use a secant approximation with
O ( N ) computing and storage time that uses second-order information.
This OSS (one-step secant) method does not require the choice of critical
parameters, is guaranteed to converge to a point with zero gradient,
and has been shown to accelerate the learning phase by many orders of
magnitude with respect to batch BP if high precision in the output values
is desired (Battiti and Masulli 1990). In cases where approximated output
values are sufficient, the OSS method is usually better or comparable with
"fair" versions of backpropagation.*'
While the term OSS should be preferred, historically OSS is a variation
of what is called one-step (memory-less) Broyden-Fletcher-Goldfarb-Shanno
method. In addition to reducing both the space and computational com-
plexity of the BFGS method to O ( N ) ,this method provides a strong link
between secant methods and the conjugate gradient methods described in
Section 3.

2aUpdatingthe Cholesky factorization and calculating the solution are both of order
O(fl).The same order is obtained if the inverse Hessian is updated, as in equation 6.1,
and the search direction is calculated by a matrix-vector product.
*'The comparison with BP with fixed learning and momentum rate has little meaning:
if an improper learning rate is chosen, standard BP becomes arbitrarily slow or not
convergent, if parameters are chosen with a slow trial-and-error process this time should
also be included in the total computing time.
First- and Second-OrderMethods for Learning 157

The derivation starts by inverting equation 5.5, obtaining the positive

definite secant update for the inverse Hessian:

(6.1)

Now, there is an easy way to reduce the storage for the matrix H, : just
forget the matrix and start each time from the identity I. Approximating
equation 6.1 in this way, and multiplying by the gradient g, = VE(w,),
the new search direction p+ becomes

(6.2)

where the two scalars A, and B, are the following combination of scalar
products of the previously defined vectors s,, g,, and ye (last step, gradient
and difference of gradients):

The search direction at the beginning of learning is taken as the nega-

tive gradient and it is useful to restart the search direction to -g, every N
steps (Nbeing the number of weights in the network). It is easy to check
that equation 6.2 requires only O ( N ) operations for calculating the scalar
products pTg,, pryc, yrgc and yFyc. Remembering that the search direction
has to be used for a fast one-dimensional search (see Section 4.1), the to-
tal computation per cycle (per epoch) required by the method is a small
multiple (2-4) of that required by one cycle of gradient descent with a
fixed learning rate.
Now, if exact line searches are performed, equation 6.2 produces mutu-
ally conjugate directions (Shanno 1978). The difference with other forms
of the conjugate gradient method is that the performance of the one-step
positive definite secant update maintains the "safety" properties even when
the search is executed in a small number of one-dimensional trials.=
While the above method is suitable for batch learning, a proposal for
a learning method usable in the on-line procedure has been presented in
LeCun (1989)and Becker and LeCun (1989). The Hessian is approximated
with its diagonal part, so that the matrix-multiplication of the gradient by
the inverse Hessian (see Newton's method) is approximated by dividing
each gradient component gn by a running estimate of hnn (ef
inn).

221f the problem is badly scaled, for example, if the typical magnitude of the
variables changes a lot, it is useful to substitute the identity matrix with Ha =
max{ lE(wo)l,typical size of E} . D:, where D, is a diagonal scaling matrix, such that
the new variables ti = D,w are in the same range [see Dennis and Schnabel(1983)l.
158 Roberto Battiti

At each iteration a particular weight w, is updated according to the

following rule23:

(6.3)

The estimate Grin of the diagonal component of the Hessian is in turn ob-
tained with an exponentiallLweighted average of the second derivative
(or an estimate thereof d2E/dw$),as follows:

(6.4)

Suppose that the weight wnconnects the output of unit j to unit i (w,=
in the double-index notation), a, is the total input to unit i, f ( ) is the
wI,,
squashing function and xI is the state of unit j . It is easy to derive

The term d2E/i)af is then computed explicitly with a backpropagation-

type procedure, as follows:

(6.6)

Finally, for the simulations in LeCun (19891, the term in equation 6.6
with the second derivative of the squashing function is neglected, as
in the Levenberg-Marquardt method, that will be described in Section 7,
obtaining

(6.7)

Note that a positive estimate is obtained in this way (so that the neg-
ative gradient is multiplied by a positive-definite diagonal matrix).
The parameters p and E in equation 6.3 and y in equation 6.4 are fixed
and must be appropriately chosen by the user. The purpose of adding
p to the diagonal approximation is explained by analogy with the trust-
region method (see equation 4.9). According to Becker and LeCun (1989)
the method does not bring a tremendous speed-up, but converges reliably
without requiring extensive parameter adjustments.

231nthis rule we omit details related to weight sharing, that is, having more connections
controlled by a single parameter.
First- and Second-Order Methods for Learning 159

7 Special Methods for Least Squares

If the error function that is to be minimized is the usual E = EL=lCyl:

(opi - tp,)2,learning a set of examples consists in solving a nonlinear least-
squares problem, for which special methods have been devised. Two of
these methods are now described: the first (the Gauss-Newton method) is
based on simplifying the computation of second derivatives, the second
(the Levenberg-Marquardt method) is a trust-region modification of the
former.
Let's define as R ( x ) the vector24 whose components are the residuals
for the different patterns in the training set and output units [rpi(w)=
(op,(w)- f p l ) ] , so that the error can be expressed as E = i R ( z ~ ) ~ R ( It
w )is.
straightforward to see that the first and second derivatives of E(w)are,
respectively:

VE(w) =

V2E(w) =

= ~ + S(w)
J ( w ) J(w) (7.2)
where I ( w )is the Jacobian matrix ] ( x ) ~ ~=
, , ar,,(w)/dw, and S(w)is the part
of the Hessian containing second derivatives of rp1(w),that is, S ( w ) =
EL rpl(w) V2rp1(w).
The standard Newton iteration is the following:
w+ = wc - V(Wc)TI(wc) + s(wc)]-l I(wc)TR(wc) 17.3)
The particular feature of the problem is that, while I(wc) is easily cal-
culated, S(wc) is not. On the other hand, a secant approximation seems
"wasteful" because part of the Hessian [the I(W~)~J(W,) part1 is easily ob-
tained from ](w)and, in addition, the remaining part S(w) is negligible
for small values of the residuals. The Gauss-Newton method consists in
neglecting the S ( w ) part, so that a single iteration is
w+ = wc - u(Wc)TI(Wc)I-' I M TR(wc) (7.4)
It can be shown that this step is completely equivalent to minimizing the
error obtained from using an affine model of R(w)around wc:

(7.5)

(7.6)
24Thecouples of indices ( p , i) can be alphabetically ordered, for example, and mapped
to a single index.
160 Roberto Battiti

The QR factorization method can be used for the solution of equation 7.5
[see Dennis and Schnabel (198311. The method is locally q-quadratically
convergent for small residuals. If J(w,) has full column rank, J ( W ~ ) ~ J ( W ~ )
is nonsingular, the Gauss-Newton step is a descent direction and the
method can be modified with line searches (dumped Gauss-Newton
method).
Another modification is based on the trust-region idea (see Section 4.3)
and known as the Levenberg-M~rquardt method. The step is defined as
w+ = wc - u(wc)TJ(wc)+ /d-
J(WJTR(WC) (7.7)
This method can be used also if J ( w )does not have full column rank (this
happens, for example, if the number of examples is less than the number
of weights).
It is useful to mention that the components of the Jacobian matrix
arpj(w)/8wIcan be calculated by the usual chain rule for derivatives
with a number of backpropagation passes equal to the number of output
units. If weight w,b connects unit b to unit a (please note that now the
usual two-index notation is adopted), one obtains

where the term S,,,, is defined as S,,,a = [Or,,(w)/~netpfl] and X p b is the

output of unit b. For the output layer, 6,,,@= f(netpe)if i = a, 0 otherwise.
For the other layers, 6,,,@= f(netPR)C, I j g r , c ~ c R , summing over the units of
the next layer (the one closer to the output).
Software for variants of the Levenberg-Marquardt method can be found
in Press et al. (1988). Other versions are MINPACK and NL2SOL (stan-
dard nonlinear least-squares packages).25 Additional techniques that are
usable for large-scale least-squares problems are presented in Toint (1987).
They are based on adaptive modeling of the objective function and have
been used for problems with u p to thousands of variables.2b Additional
references are Gawthrop and Sbarbaro (1990)and Kollias and Anastassiou
(1989). In Kollias and Anastassiou (1989) the Levenberg-Marquardt tech-
nique is combined with the acceleration techniques described in Jacobs
(1988) and Silva and Almeida (1990).

8 Other Heuristic Strategies

Some learning methods have been introduced specifically for backprop-

agation that show promising performance on some tests problems.
251 owe this information to Prof. Christopher G. Atkescm. MINPACK is described in
More e t a / . (1980) and NLZSOL in Dennis ef al. (1981).
261nreality the test problems presented in this paper have a special partially separa-
ble structure, so that their practical application to multilayer perceptrons with complete
connectivity is still a subject of research.
First- and Second-OrderMethods for Learning 161

Because the standard algorithm involves selecting appropriate learn-

ing and momentum rates, it is convenient to consider ways to adapt these
parameters during the search process. In this case the trial-and-error se-
lection is avoided and, in addition, the possibility to tune the parameter
to the current properties of the error surface usually yields faster con-
vergence with respect to using fixed coefficients.
An heuristic method for modifying the learning rate is, for example,
described in Lapedes and Farber (19861, Vogl et al. (1988), and Battiti
(1989) (the bold driver (BD) method). The idea is to increase the learning
rate exponentially if successive steps reduce the error, and decrease it
rapidly if an accident is encountered (increase of the error), until a
proper rate is found (if the gradient is significantly different from zero,
letting the step go to zero will eventually decrease the error). After
starting with a small learning its modifications are described by
the evolution equation:

if E [ w ( t ) ]< E[w(t- l)]

E(t)
if E [ w ( t ) ]2 E [ w ( t- l)]using t ( t - 1) (8.1)
=
01 E(t - 1)

where p is close to one (say p = 1.1)in order to avoid frequent accidents

(because the error computation is wasted in these cases), 0 is chosen to
provide a rapid reduction (say (T = 0.5), and l is the minimum integer
such that the reduced rate [ d E ( f - 1)l succeeds in diminishing the error.
The performance of this quick and dirty version is close and usually
better than the one obtained by appropriately choosing a fixed learning
rate in batch BP.
Suggestions for adapting both the search direction and the step along
this direction are presented in Chan and Fallside (1987) and Jacobs (1988).
In Chan and Fallside (1987)the learning and momentum rates are adapted
to the structure of the error surface, by considering the angle O k between
the last step and the gradient direction and by avoiding domination
of the weight update by the momentum term (in order to avoid ascent
directions). The weight update is:

where

A comparison of different techniques is presented in Chan (1990). In

Jacobs (1988) each individual weight has its own learning rate, that is
modified in order to avoid oscillations. In the proposed delta-bar-delta

271f the initial rate is too large, some iterations are wasted to reduce it until an ap-
propriate rate is found.
162 Roberto Battiti

method, the learning rate modification is the following:

if 8(t - 1 ) 6 ( t ) > 0
if 8(t - 1 ) 6 ( t ) < o (8.3)
otherwise
where
-
6 ( t ) = (1 - e)s(t) + e q t - 1)

The current partial derivative with respect to a weight is compared with

an exponentially weighted average s(t) of the previous derivatives. If
the signs agree, the (individual) learning rate E is increased by K if they
disagree (a symptom of oscillations) it is decreased by a portion 4 of
its value. A similar acceleration technique has been presented in Silva
and Almeida (1990). An heuristic technique (quickprop) loosely based
on Newtons method is presented in Fahlman (1988). In quickprop each
weight is updated in order to reach the minimum of an independent
quadratic model (parabola).

9 Summary and Conclusions

A review of first- and second-order methods suitable for learning has

been presented. Modifications of first-order backpropagation and second-
order methods are currently used by many researchers in different fields,
both to obtain faster convergence times and to avoid the meta-optimization
phase (where the user has to select proper parameters for learning, check
the numerical stability, and, in general, optimize the performance of the
learning method). This second point has a considerable impact in order
to maintain, and possibly improve, the short development times required
by neural networks with respect to methods based on intensive knowl-
edge engineering.
In the previous sections some of these techniques have been refer-
enced, with particular emphasis on their local and global convergence
properties, numerical stability and memory /computation requirements.
Some second-order techniques require a large amount of computa-
tion per iteration28[of order O(N2) or O(N3)l and/or large amounts of
memory [of order O ( P ) l . Nonetheless, they are still applicable to prob-
lems with a limited number of weights (say < 100) and show superior
performance with respect to standard backpropagation, especially if high
precision in the input-output mapping function executed by the network
is required. For problems with more weights, suitable approximations
can be applied, obtaining methods with O ( N )behavior, while still main-
taining some safety and progress properties of their parent methods.
=It is clear that, in order to obtain the total learning time, the estimates must be
multiplied by the number of patterns P and the total number of iterations. In this
paper these factors are omitted for clarity of exposition.
First- and Second-Order Methods for Learning 163

While the presentation has focused onto the multilayer perceptron neural
network, most of these techniques can be applied to alternative m0dels.2~
It is also worth stressing that problems related to memory require-
ments are less stringent now than when these methods were invented,
and problems related to massive computation can be approached by us-
ing concurrent computation. Most of the presented techniques are suitable
for a parallel implementation, with a speed-up that is approximately
proportional to the number of processors employed [see, for example,
Kramer and Sangiovanni-Vicentelli (1988); Battiti et al. (1990); Battiti and
Straforini (199111.
We feel that the cross-fertilization between optimization techniques
and neural networks is fruitful and deserves further research efforts. In
particular the relevance of second-order techniques to large-scale back-
propagation tasks (with thousands of weights and examples) is a subject
that deserves additional studies and comparative experiments.

Acknowledgments

The author is indebted to Profs. Geoffrey Fox, Roy Williams, and Edoardo
Amaldi for helpful discussions. Thanks are due to Profs. Tommaso Pog-
gio, Christopher Atkeson, and Michael Jordan for sharing their views on
second-order methods. The results of a second-order survey by Eric
A. Wan (Neuron Digesf 1989, 6-53) were a useful source of references.
The referees detailed comments are also greatly appreciated. Part of this
work was completed while the author was a research assistant at Cal-
tech. The research group was supported in part by DOE Grant DE-FG-
03-85ER25009, The National Science Foundation with Grant IST-8700064,
and by IBM.

References

Allred, L. G., and Kelly, G. E. 1990. Supervised learning techniques for back-
propagation networks. Proc. Int. Joint Conf. Neural Networks (IJCNN), Wash-
ington I, 721-728.
Barnard, E., and Cole, R. 1988. A neural-net training program based on conju-
gate gradient optimization. Oregon Graduate Center, CSE 89-014.
Battiti, R. 1989. Accelerated back-propagationlearning: Two optimization meth-
ods. Complex Syst. 3, 331-342.
Battiti, R., and Masulli, F. 1990. BFGS optimization for faster and automated
supervised learning. Proc. Int. Neural Network Conf. (INNC go), Paris, France
757-760.

29For example the RBF model introduced in Broomhead and Lowe (1988) and Poggio
and Girosi (1990).
164 Roberto Battiti

Battiti, R., and Straforini, M. 1991. Parallel supervised learning with the mem-
oryless quasi-Newton method. In Parallel Computing: Problems, Methods and
Applications, P. Messina and A. Murli, eds. Elsevier, Amsterdam.
Battiti, R., Colla, A. M., Briano, L. M., Cecinati, R., and Guido, P. 1990. An
application-oriented development environment for neural net models on the
multiprocessor Emma-2. Proc. IFIP Workshop Silicon Architectures Neural Nets,
St Paul de Vence, France, M. Somi and J. Calzadillo-Daguerre, eds. North-
Holland, Amsterdam.
Becker, S., and LeCun, Y. 1989. Improving the convergence of backpropagation
learning with second-order methods. In Proceedings of the 1988 Connection-
ist Models Summer School, D. Touretzky, G. Hinton, and T. Sejnowski, eds.
Morgan Kaufmann, San Mateo, CA.
Bengio, Y., and Moore, B. 1989. Acceleration of learning. Proc. GNCB-CNR
School, Trento, Italy.
Bingham, J. A. C. 1988. The Theory and Practice of Modem Design. Wiley, New
York.
Broomhead, D. S., and Lowe, D. 1988. Multivariable functional interpolation
and adaptive networks. Complex Syst. 2,321-355.
Broyden, C. G., Dennis, J. E., and More, J. J. 1973. On the local and superlinear
convergence of quasi-Newton methods. J.I.M.A. 12, 223-246.
Cardon, H., van Hoogstraten, R., and Davies, P. 1991. A neural network ap-
plication in geology: Identification of genetic facies. Proc. Int. Conf. Artificial
Neural Networks (ICANN-91),Espoo, Finland 1519-1522.
Chan, L. W. 1990. Efficacy of different learning algorithms of the backpropaga-
tion network. Proc. I E E E TENCON-90.
Chan, L. W., and Fallside, F. 1987. An adaptive training algorithm for back
propagation networks. Cornput. Speech Language 2,205-218.
Dennis, J. E., and Schnabel, R. B. 1983. Numerical Methods for Unconstrained
Optimization and Nonlinear Equations. Prentice Hall, Englewood Cliffs, NJ.
Dennis, J. E., Gay, D. M., and Welsch, R. E. 1981. Algorithm 573 NL2SOL -
An adaptive nonlinear least-squares algorithm [E4]. TOMS 7,369-383.
Dongarra, J. J., Moler, C. B., Bunch, J. R., and Stewart, G. W. 1979. LINPACK
Users's Guide. Siam, Philadelphia.
Drago, G. P., and Ridella, S. 1991. An optimum weights initialization for im-
proving scaling relationships in BP learning. Proc. Int. Conf. Artificial Neural
Networks (ICA"-91 ), Espoo, Finland 1519-1 522.
Fahlman, S. E. 1988. An empirical study of learning speed in back-propagation
networks. Preprint CMU-CS-88-162, Carnegie Mellon University, Pittsburgh,
PA.
Gawthrop, I?, and Sbarbaro, D. 1990. Stochastic approximation and multilayer
perceptrons: The gain backpropagation algorithm. Complex Syst. 4,51-74.
Gill, P. E., Murray, W., and Wright, M. H. 1981. Practical Optimization. Academic
Press, London.
Glover, F. 1987. TABU Search methods in artificial intelligence and operations
research. ORSA Art. Int. Nezuslett. l(2.6).
Goldfarb, D. 1976. Factorized variable metric methods for unconstrained opti-
mization. Math. Comp. 30,796-811.
First- and Second-Order Methods for Learning 165

Goldstein, A. A. 1967. Constructive Real Analysis. Harper & Row, New York, NY.
Jacobs, R. A. 1988. Increased rates of convergence through learning rate adap-
tation. Neural Networks 1, 295-307.
Johansson, E. M., Dowla, F. U., and Goodman, D. M. 1990. Backpropagation
learning for multi-layer feed-forward neural networks using the conjugate
gradient method. Lawrence Livermore National Laboratory, Preprint UCRL-
JC-104850.
Kollias, S., and Anastassiou, D. 1989. An adaptive least squares algorithm for
the efficient training of multilayered networks. I E E E Trans. CAS 36, 1092-
1101.
Kramer, A. H., and Sangiovanni-Vicentelli, A. 1988. Efficient parallel learning
algorithms for neural networks. In Advances in Neural Information Processing
Systems, Vol. 1, pp. 75-89. Morgan Kaufmann, San Mateo, CA.
Lapedes, A., and Farber, R. 1986. A self-optimizing, nonsymmetrical neural
net for content addressable memory and pattern recognition. Physica 22 D,
247-259.
LeCun, Y. 1986. HLM A multilayer learning network. Proc. 2986 Connectionist
Models Summer School, Pittsburgh 169-177.
LeCun, Y. 1989. Generalization and network design strategies. In Connectionism
in Perspective, pp. 143-155. North Holland, Amsterdam.
LeCun, Y., Kanter, I., and Solla, S. A. 1991. Second order properties of error
surfaces: Learning time and generalization. In Neural Information Processing
Systems - NIPS, Vol. 3, pp, 918-924. Morgan Kaufmann, San Mateo, CA.
Luenberger, D. G. 1973. Introduction to Linear and Nonlinear Programming. Addi-
son-Wesley, New York.
Luo, Zhi-Quan. 1991. On the convergence of the LMS algorithm with adaptive
learning rate for linear feedforward networks. Neural Comp. 3, 227-245.
Malferrari, L., Serra, R., and Valastro, G. 1990. Using neural networks for signal
analysis in oil well drilling. Proc. 111Ital. Workshop Parallel Architect. Neural
Networks - Vietri s/m Salerno, Italy 345-353. World Scientific, Singapore.
Merller, M. F. 1990. A scaled conjugate gradient algorithm for fast supervised
learning. PB-339 Preprint. Computer Science Department, University of
Aarhus, Denmark. Neural Networks, to be published.
MorP, J. J., Garbow, B. S., and Hillstrom, K. E. 1980. User guide for MINPACK-1.
Argonne National Labs Report ANL-80-74.
Nocedal, J. 1980. Updating quasi-Newton matrices with limited storage. Math.
Comp. 35, 773-782.
Parker, D. B. 1987. Optimal algorithms for adaptive networks: Second-order
back propagation, second-order direct propagation, and second-order Heb-
bian learning. Proc. ICNN-1, San Diego, C A 11-593-11-600.
Poggio, T., and Girosi, F. 1990. Regularization algorithms for learning that are
equivalent to multilayer networks. Science 247, 978-982.
Press, W. H., Flannery, B. I?, Teukolsky, S. A., and Wetterling, W. T. 1988. Nu-
merical Recipes in C. Cambridge University Press, Cambridge.
Rumelhart, D. E., and McClelland, J. L. (eds.) 1986. Parallel DistributedProcessing:
Explorations in the Microstructure of Cognition. Vol. 1: Foundations. MIT Press,
Cambridge, MA.
166 Roberto Battiti

Sejnowski, T. J., and Rosenberg, C. R. 1986. NETtalk: A parallel network that

learns to read aloud. The Johns Hopkins University EE and CS Tech. Rep.
JHU/ EECS-86 / 01.
Shanno, D. F. 1978. Conjugate gradient methods with inexact searches. Math.
Oper. Res. 3-3, 244-256.
Silva, F., and Almeida, L. 1990. Acceleration techniques for the backpropagation
algorithm. Lecture Notes in Computer Science, Vol. 412, pp. 110-119. Springer-
Verlag, Berlin.
Toint, L. 1987. On large scale nonlinear least squares calculations. SlAh4 I . Sci.
Stat. Cornput. 8(3), 416-435.
Vincent, J. M. 1991. Facial feature location in coarse resolution images by multi-
layered perceptrons. PYOC.lnt. Conf. Artificial Neuraf Networks (ICANN-%I,
Espoo, Finland 821-826.
Vogl, T. P., Mangis, J. K., Rigler, A. K., Zink, W. T., and Alkon, D. L. 1988. Ac-
celerating the convergence of the back-propagation method. Biol. Cybernet.
59, 257-263.
Watrous, R. 1987. Learning algorithms for connectionisf networks: Applied gradi-
ent methods of nonlinear optimization. Tech. Rep. MS-CIS-87-51, University of
Pennsylvania.
Webb, A. R., Lowe, D., and Bedworth, M. D. 1988. A comparison of nonlinear
optimization strategies for adaptive feed-forward layered networks. RSRE
MEMO 4157, Royal Signals and Radar Establishment, Malvern, England.
White, H. 1989. Learning in artificial neural networks: A statistical perspective.
Neural Corny. 1,425464.
Widrow, B., and Stearns, S. D. 1985. Adaptive Signal Processing. Prentice Hall,
Englewood Cliffs, NJ.
Williams, P. M. 1991. A Morquordt algorithm for choosing the step-size in
backpropagation learning with conjugate gradients. Preprint of the School
of Cognitive and Computing Sciences, University of Sussex (13 February
1991).

~~

Received 28 December 1990; accepted 13 September 1991.