Shrinkage regression

Rolf Sundberg Volume 4, pp 19941998 in Encyclopedia of Environmetrics (ISBN 0471 899976) Edited by Abdel H. El-Shaarawi and Walter W. Piegorsch
John Wiley & Sons, Ltd, Chichester, 2002

Shrinkage regression
Shrinkage regression refers to shrinkage methods of estimation or prediction in regression situations, useful when there is multicollinearity among the regressors. With a term borrowed from approximation theory, these methods are also called regularization methods. Such situations occur frequently in environmetric studies, when many chemical, biological or other explanatory variables are measured, as when Branco et al. [3] wanted to model toxicity of n D 19 industrial wastewater samples by pH and seven metal concentrations, and found it necessary to use dimensionality-reducing techniques. As another example, to be discussed further below, Andersson et al. [1] constructed a quantitative structureactivity relationship (QSAR) to model polychlorinated biphenyl (PCB) effects on cell-to-cell communication inhibition in the liver, having a sample of size n D 27 of PCB congeners that represented a set of 154 different PCBs, and using 52 physicochemical properties of the PCBs as explanatory variables. Suppose we have n independent observations of x, y D x1 , . . . , xp , y from a standard multiple regression model Y D C 0 x C 1

unrealistically large values. Under exact collinearity, bOLS is not even uniquely dened. In these situations it can pay substantially to use shrinkage methods that trade bias for variance. Not only can they give more realistic estimates of , but they are motivated even stronger for construction of a predictor of y from x. A simple and as we will see fundamental shrinkage construction is ridge regression (RR), according to which the OLS estimator is replaced by bRR D Sxx C I
1

sxy

var D 2 (see Linear models). The ordinary least squares (OLS) estimator can be written bOLS D Sxx1 sxy 2

where Sxx is the sum of squares and products matrix of the centered x variables and sxy is the vector of their sums of products with y. The OLS estimator is the best tting and minimum variance linear unbiased estimator (best linear unbiased estimator, BLUE), with variance var bOLS D
2

Sxx1

That OLS yields the best t does not say, however, that bOLS is best in a wider sense, or even a good choice. We will discuss here alternatives to be used when the x variables are (near) multicollinear, that is when there are linear combinations among them that show little variation. The matrix Sxx is then near singular, so var bOLS will have very large elements. Correspondingly, the components of bOLS may show

for a suitable ridge constant > 0 (to be specied). Addition of whatever > 0 along the diagonal of Sxx clearly makes this factor less close to singular. To see more precisely how RR shrinks, consider an orthogonal transformation that replaces the components of x by the principal components (PCs), ti D ci x, i D 1, . . . , p. Here the coefcient vectors ci are the eigenvectors of Sxx , with eigenvalues i , say. Correspondingly, Sxx is replaced by the diagonal matrix of these eigenvalues i , and RR adds to them all. When OLS divides the ith component of the vector sty by i , RR divides by i C . Hence, RR shrinks bOLS much in principal directions with small , but only little in other principal directions. Variable selection might appear to shrink, by restricting attention to a lower-dimensional subspace Ur r < p of the model space of the xs. Variable selection improves precision when it omits variables with negligible inuence, but if it is used to eliminate multicollinearities, then it does not necessarily shrink and will make interpretation difcult. More sophisticated ways of restricting attention to lower-dimensional subspaces Ur of regressors are represented by methods such as principal components regression (PCR) and partial least squares (or projection to latent structures; PLS) regression. Both methods construct new regressors, linear in the old regressors, and y is regressed (by least squares, LS) on a relatively small number of these PCs or PLS factors. This number r, as well as the ridge constant in RR, is typically chosen by a cross-validation criterion. As an example, Andersson et al. [1] used PLS regression with two PLS factors to replace the 52 original variables in the regression. This model explained R2 D 84% of the variability, and in a crossvalidation predicted 80% (necessarily less than R2 , but quite close in this case). They could also interpret their model within the context of their chemometric

Shrinkage regression
The most important role of CR is perhaps not as a practical regression method, however, but as a theoretical device embracing several methods. For D 0 we get back OLS, and the algorithm will stop after one step r D 1 . For D 1 the criterion function (5) is proportional to the covariance squared and will yield PLS, and as ! 1, V t will dominate and we get PCR. A much more general yet simple result, which relates to RR and provides an even wider understanding, has been given in [2], as follows. Suppose we are given a criterion that is a function of R2 t, y and V t , nondecreasing in each of these arguments. The latter demand is quite natural, because we want both of these quantities to have high values. The regressor maximizing any such criterion under the constraint jcj D 1 is then proportional to the RR estimator (4) for some , possibly negative or innite. Conversely, for each 0 or < max in (4) there is a criterion for which this is the solution. The rst factor PLS is obtained as ! 1, and ! max corresponds to the rst factor PCR. The rst factor CR with varying 0 usually (but not always) requires the whole union of 0 and < max . The use of c D bRR from (4) to form a single regressor in LS regression is called least squares ridge regression (LSRR) in [2], and differs from RR by a scalar factor that will be near 1 when is small. Hence RR is closely related to the other constructions. Furthermore, experience shows that typically RR and LSRR for some best small yield a similar regression as (the best r) PLS and (best r) PCR. If different constructions yield quite similar estimators/predictors, what shrinkage method is then preferable? Here is some general advice. 6 7 RR and LSRR have the advantage that they only involve a single factor. LSRR only shrinks to compensate for multicollinearity, whereas RR always shrinks. RR has a Bayesian interpretation as posterior mean for a suitable prior for (see Bayesian methods and modeling). LSRR (like PLS and PCR) yields residuals orthogonal to the tted relation, unlike RR. PLS and PCR have interpretations in terms of latent factors, such that for PCR essentially all x variability is in Ur for suitable r, whereas for PLS all x variability that inuences y is in the corresponding Ur for a typically somewhat smaller r than for PCR.

application, as referring basically to the position pattern of the chlorine atoms. In the standard form of PCR, the new regressors ti D ci x are the rst PCs of the (centered) x from a PCs analysis. This means that the ci s are the eigenvectors of Sxx that explain as much variation as possible in x. The space Ur is then spanned by the rst r PCs. In this way directions in x with little variation are shrunk to 0. The connection with variation in y is only indirect, however. PLS regression is more efcient in this respect, because it constructs a rst regressor t1 D c1 x to maximize covariance between t1 and y (among coefcient vectors c1 of xed length), and then successively t2 , t3 , etc. so as to maximize covariance with the y residuals after LS regression of y on the previous set of regressors. As an algorithm for construction of the PLS regression subspaces Ur of the x space, spanned by t1 , . . . tr , this is one of several possible. For a long time PLS was only algorithmically dened, until Helland [10] gave a more explicit characterization by showing that Ur for PLS is spanned by the sequence of vectors sxy , Sxx sxy , . . . , Sr 1 sxy . This representation is of more xx theoretical than practical interest, however, because there are numerically better algorithms. A step towards a general view on the choice of shrinkage regression was taken by Stone and Brooks [13], who introduced continuum regression (CR), with the criterion function R2 t, y V t 5

where R t, y is the sample correlation between t D c x and y, V t is the sample variance of t, and 0 is a parameter to be specied. This function of t (or c) should be maximized under the constraint jcj D 1. In previous terms R2 t, y / c sxy 2 c Sxx c

V t / c Sxx c

As in PLS, when a rst c D c1 , t D t1 , has been found, the criterion is applied again on the residuals from LS regression of y on t1 to nd the next c, c D c2 , etc. The new c will automatically be orthogonal to the previous ones. In order to institute a regression method of this algorithm, the number r of regressors and the parameter are chosen so as to optimize some cross-validation measure.

Shrinkage regression
The latent factor structure in PCR and PLS is convenient for outlier detection and classication (cf. below and [12, Chapter 5]). Also x components missing at random in prediction of y for a new observation are easily handled.

Multicollinearity imposes estimation identiability problems, when we want to nd the true . If x1 C x2 is (near) constant, say, we cannot tell if x1 or x2 or both jointly are responsible for the inuence of x on y. This is less serious for prediction, provided that new observations satisfy the same relation between y and x, and that new xs are like calibration xs, in particular satisfying the same multicollinearity. To judge the predictive ability, some version of crossvalidation is natural, based on repeated splits of data in calibration and test sets. But we must always have in mind the risk that the calibration (x, y) data are particular in some respect, and that therefore crossvalidation criteria are too optimistic. An extreme example, from a study concerning determination of nitrogen in wastewater, is discussed in [14]. In the QSAR example above [1], the model was tted on a sample of 27 PCBs from a population group of 154 PCBs. If we want to predict the activity y for a new PCB from this population, then we must allow and check for the possibility that it is an outlier in x space, compared with the calibration sample. Implications differ according to whether it is an outlier in a direction orthogonal to the two-dimensional latent subspace Ur , or only within Ur . There are also other PCBs not even satisfying the criteria for belonging to the population group under study, and for them we must doubt the prediction equation for y even if x is not an outlier. Shrinkage regression constructions are intrinsically scale noninvariant. This is because near in near-collinear is metric-dependent. Correlation is invariant, but not covariance and variance. Individual rescaling of the x components, or any other nonorthogonal transformation, will affect the shrinkage. It is often recommended that x components be autoscaled to unit variances as a pretreatment. This is natural when the x components are on incomparable scales, as in QSAR studies, but silly in many other situations, for example in spectroscopy. As another example of frequent pretreatments, secondorder differencing of a spectrum x corresponds to a nonorthogonal linear transformation, and thus has

effects on shrinkage regression. The choice of scaling is often more crucial than the choice between the different methods (RR, LSRR, PCR, PLS). In multivariate regression, when the response is a vector y and not a scalar y, and the regression coefcients form a matrix B, we could try shrinkage also in y space. In this way we might be able to borrow strength for the prediction of one component of y from the others. A variety of methods has been proposed in the literature. There are PLS type algorithms (PLS2, SIMPLS), in each step constructing a pair of PLS factors, in y and in x. The rst factors are selected equally by PLS2 and SIMPLS, but their criteria for the subsequent factors differ. There is a multivariate version of CR, joint continuum regression (JCR [5]), of which SIMPLS is a special case. At one of the extreme ends of JCR we nd PCR, which is not affected by the dimension of y. At the other end is the LS version of reduced-rank regression (RRR). RRR, imposing a rank constraint on B, is an example of a principle that essentially only shrinks in y space and therefore is inadequate under near-collinearity in x. The maximum likelihood (ML) (6D LS) estimator in an RRR model (standard multivariate regression with a xed rank constraint added) is equivalent to using a number of canonical variates from a canonical correlation analysis, so for each canonical y variate we use OLS regression. This method has become an important tool in econometrics for modeling cointegration (i.e. common trends) in nonstationary vector autoregressive time series models [11], with potential applications also in environmetrics. Burnham et al. [8] formulated a latent variable multivariate regression model, in which both y and x have latent structures, more or less overlapping. The overlap yields a regression relationship, when y should be predicted from x. A constrained ML method in this model is equivalent to a method known as principal covariates regression (PcovR [9]), which is a shrinkage method that will often yield similar results to those of PLS2 and SIMPLS. See also [7], giving characterizations and relationships between several different methods with respect to their optimality criteria of construction. Further methods were discussed and compared by Breiman and Friedman [4], who constructed and advocated a method they called curds and whey, primarily shrinking in y space. The article caused lively debate and much criticism.

Shrinkage regression
[4] Breiman, L. & Friedman, J.H. (1997). Predicting multivariate responses in multiple linear regression (with discussion), Journal of the Royal Statistical Society, Series B 59, 354. Brooks, R.J. & Stone, M. (1994). Joint continuum regression for multiple predictands, Journal of the American Statistical Association 89, 13741377. Brown, P.J. (1993). Measurement, Regression, and Calibration, Oxford University Press, Oxford. Burnham, A.J., Viveros, R. & MacGregor, J.F. (1996). Frameworks for latent variable multivariate regression, Journal of Chemometrics 10, 3145. Burnham, A.J., MacGregor, J.F. & Viveros, R. (1999). A statistical framework for multivariate latent variable regression methods based on maximum likelihood, Journal of Chemometrics 13, 4965. De Jong, S. & Kiers, H.A.L. (1992). Principal covariates regression. Part I. Theory, Chemometrics and Intelligent Laboratory Systems 14, 155164. Helland, I.S. (1988). On the structure of partial least squares regression, Communications in StatisticsSimulation and Computation 17, 581607. Johansen, S. (1995). Likelihood-based Inference in Cointegrated Vector Autoregressive Models, Oxford University Press, Oxford. Martens, H. & Ns, T. (1989). Multivariate Calibration, Wiley, Chichester. Stone, M. & Brooks, R.J. (1990). Continuum regression: cross-validated sequentially constructed prediction embracing ordinary least squares, partial least squares and principal components regression (with discussion), Journal of the Royal Statistical Society, Series B 52, 237269. Sundberg, R. (1999). Multivariate calibration direct and indirect regression methodology (with discussion), Scandinavian Journal of Statistics 26, 161207.

Despite the many shrinkage methods proposed and used for multivariate regression, it is not clear if and when they are really better than separate univariate regressions. From the literature it seems as if one seldom gains much precision from replacing separate univariate predictions by a multivariate prediction method, cf. the discussion in [5]. For fully implemented PCR or PLS, with crossvalidation to help select the number of factors and various diagnostic tools, there are commercial packages from the eld of chemometrics doing the job, such as SIMCA (Umetrics) and UNSCRAMBLER (CAMO). For those who want more exibility or more lucid software, programming in Matlab can be recommended. The PLS Toolbox (Eigenvector Research) contains a large number of Matlab subroutines, and there is also much free code available on the web. Recommended literature on shrinkage regression includes the books by Brown [6], not least for covering Bayesian aspects, and Martens and Ns [12].

[5]

[6] [7]

[8]

[9]

[10]

[11]

References
[1] Andersson, P., Haglund, P., Rappe, C. & Tysklind, M. (1996). Ultra violet absorption characteristics and calculated semi-empirical parameters as chemical descriptors in multivariate modelling of polychlorinated biphenyls, Journal of Chemometrics 10, 171185. Bj rkstr m, A. & Sundberg, R. (1999). A generalized o o view on continuum regression, Scandinavian Journal of Statistics 26, 1730. Branco, J.A., Pires, A.M., Mendon a, E. & Picado, c A.M. (1994). A statistical evaluation of toxicity tests of waste water, in Statistics for the Environment, Vol. 2, V. Barnett & K.F. Turkman, eds, Wiley, Chichester.

[12] [13]

[2]

[14]

[3]

(See also Collinearity) ROLF SUNDBERG