7SSMM700 Lecture 8

Regularization techniques and shrinkage estimators
7SSMM700 Quantitative Methods for Finance and Data Analytics
Leone Leonida and Marina Dolfin

M.Sc. Banking and Finance
King’s Business School
King’s College London
Week 8
L. Leonida - M. Dolfin (King’s College London) M.Sc. Banking and Finance Week 8 1 / 33
The big data era is creating plenty of opportunities for new developments
in econometrics, economics, and finance.
The recent improvements in computer science are making data collection

easier.
Meanwhile, the analysis of large data sets poses methodological challenges

(for example, high-frequency observations, unstructured data, and new
large datasets).
We explore the recent advancements in macroeconometrics and empirical

financial analysis to deal with big data. See how machine learning
techniques can be implemented to estimate macro and financial data for
forecasting and structural analysis.
Tools to manage Big Data
Introduction to Regularization
Theoretical approach
Ridge Regression
Lasso Regression
Elastic Net Regression
Variable selection in time series applications
CLRM vs. ML: an example in predicting returns on the Forex with

high frequency data.
Introduction to Regularization
Consider the standard model of ordinary least squares (OLS) for multiple
linear regression
y = Xβ + e (1)
where y ∈ RN , β ∈ Rk , and X ∈ RN ×k .
Multicollinearity and Overfitting
The assumptions for the Gauss-Markov theorem are mostly true in smaller
datasets, which makes OLS a very powerful tool for statisticians and
scientists.
However, these assumptions tend to be false with sufficiently large

datasets and therefore the OLS method can cause some issues with the
resulting model.
Multicollinearity and Overfitting
One of the most common issues with the OLS method is the tendency for
the model to overfit the data when there is too much noise caused by
correlated variables. This can happen in many different situations.
The most extreme case occurs when p > n. In this case there exists no
unique solution to the system and linear regression fails to produce
accurate coefficient values.
With less extreme situations multicollinearity can cause the model to be

overly sensitive to small changes in parameter values and coefficients can
have the ”wrong” sign or an incorrect order of magnitude.
When overfitting, the coefficients can hive high standard errors and low
levels of significance despite a high R 2 value.
Ridge Regression and LASSO (Least Absolute Shrinkage and Selection)
improve the overall accuracy of ordinary least squares regression by adding
a bias that imposes shrinkage on the model that greatly reduces the
variance of coefficient estimates.
Regularization or shrinkage
Both LASSO and ridge regression estimation minimize the sum-of-squares

plus a penalty. The latter encourages the estimator to be small, in
particular closer to zero.
The penalty imposes a shrinkage on the coefficient estimates of ordinary

least squares. and its particular form yields different types of shrinkage.
Generally, for the Lp regularization term we have

1
Lp = (∑ | β i |p ) p
i
Ridge and LASSO deal with the L2 and L1 penalties respectively.
Some notation for LASSO and Ridge penalties
L1 = ∑ | βi |
i
q
L2 = ∑ β2i
i
Regularization is used in preference over other common methods of

determining the best linear model, such as best subset selection and
stepwise subset selection.
Ridge Regression
RSS
Ridge regression adds the L2 penalty such that we have
(y − X β)T (y − X β) + λβT β (2)
Ridge estimator
From equation (2) we can derive the ridge coefficient estimate:
(y − X β)T (y − X β) + λβT β
= (y T − βT X T )(y − X β) + λβT β
= y T Y − Y T X β − βT X T y + βT X T X β + λβT β
= y T Y − 2y T X β + βT X T X β + λβT β
∂
→ = 0 − 2y T X + 2X T X β + 2λβ = 0
∂β
β̂ ridge = (X T X + λI )−1 X T y (3)

Discussion of the Ridge estimator
In equations (2) and (3), λ ≥ 0 is a tuning parameter for the penalty,

which is determined separately.
When λ = 0 we have the ordinary least squares estimate, and when

λ → ∞ all the coefficients approach zero.
The selection of lambda, and thus the optimal model, will be not be
discussed here. However, there is an existence theorem that states there
always exists a λ > 0 such that the MSE is less than that of the least
squares estimate λ = 0. A proof of the theorem can be found in Hoerl
(1970).
Advantages of Ridge
Ridge regression’s advantage over ordinary least squares lies in it’s

bias-variance trade-off. As λ increases, the flexibility of the model fit
decreases. This leads to increased bias but decreased variance.
Disadvantages of Ridge
Ridge regression has a major disadvantage to other methods dealing with

ill-posed problems and overfitting: it does not perform feature selection.
While ridge shrinks coefficients towards zero, the final model chosen will
always include all of the predictors (unless λ = ∞ in which all predictors
will be zero).
LASSO estimator
The LASSO model can be shown in a similar form as for the Ridge
p
(y − X β )T (y − X β ) + λ ∑ | β j | (4)
j =1
Advantages The largest benefit of LASSO is the model’s ability to create

sparse matrices, i.e. performing feature selection.
Disadvantages The main disadvantage with respect to Ridge is that

because the L1 penalty contains absolute values, it is much more difficult
to solve analytically. Like Ridge, the correct choice of λ and thus the
optimal model is very important.
Comparing Ridge and LASSO estimators
Alternatively, one can show that the Ridge and LASSO models solve these
equations respectively:
Ridge equation
(y − X β)T (y − X β) such that βT β ≤ t (5)
LASSO equation
(y − X β)T (y − X β) such that | β| ≤ t (6)
This means that for every value of λ in Ridge and LASSO, there exists a t
such that you will get the same coefficient estimates for (2) and (5), and
the same estimates for (4) and (6). When p = 2, (5) shows that ridge
regression has the smallest RSS out of all the points that lie within the
diamond defined by | β 1 | + | β 2 | ≤ t. (6) shows LASSO performs the same
with the points that lie within the circle defined by β21 + β22 ≤ t.
Graphical representation
Figure: Image taken from 1
1 Tibshirani, R. (1996).
Graphical representation: comment
In the above figure, β̂ is the least squares solution, and the diamond and
circle portray the ridge and LASSO constraints given in (6). The ellipses
around β̂ are lines of constant RSS.
Equation (5) shows that the LASSO and ridge coefficient estimate is
where the ellipses and constraint regions meet.
Since the constraint region of ridge is a circle, the probability that the
intersection will occur on an axis is zero. In contrast, the diamond
constraint region has corners at each axis, so the intersection of the
ellipses and the constraint region will often occur on the axis.
Instabilities in the Ridge regression
Figure: If moving along the “ridge ”, large changes in the parameters estimate
cause small changes in the error, determining instabilities.
Elastic Net Regression
General penalty term

p
λ ∑ [(1 − α)| β j | + α( β j )2 ]
j =1
λ=0
OLS.
α=1
Ridge regression.
α=0
LASSO regression.
All regressors must be standardized, i.e. they have zero mean and variance
equal to 1.
Application to returns in the Forex market
Data Intra-day data, five years (January 2007 - January 2011) worth of
one-minute, on returns on the currency pair EUR/USD.
Application to returns in the Forex market
Model Linear regression model
Forecasting Forecasting future returns (60 minutes), using short-term

returns and technical indicators (15 regressors).
Tools Comparison CLRM vs. stepwise selection vs. ML by cross-validation

on in-sample and out-sample.
Set of regressors: recent returns and technical indicators
Recent returns
N-Minute returns
over windows lengths of 5, 10, 15, 20, 25, 30, 60 mins
MACD Moving Average Convergence/Divergence
RSI Relative Strength Index

over windows lengths of 5, 10, 15, 20, 25, 30, 60 mins
Regressand
Future returns (60 mins in advance)
OLS estimate
Back-testing
Look for the effectiveness of the prediction by adopting a back-testing

strategy.
We calculate the returns by simply multiplying the actual returns by the

sign of the predicted returns, then apply a cumprod to generate a returns
curve
Trading with the model Take a very simple strategy (realistically, we would
put in some minimum return before we actually trade): whenever the
returns prediction is greater than 0, go long, and whenever the returns
prediction is less then 0, go short
Figure: Backtesting the OLS estimate
Repeat this process using a stepwise algorithm to discard terms and retain
only those predictors with the most predictive capability.
The algorithm remove all those factors that are less statistically significant,
returning a more compact model.
Stepwise estimate
Backtesting the OLS and stepwise estimates
Comparison with the Elastic Net regularization
Conclusions
The Elastic Net Regularization is outperforming on the out-sample both

the OLS and the stepwise, even if robustness with respect to the splitting
of the data should be checked before reaching a definite conclusion.
Readings
Tibshirani, R. (1996) “Regression Shrinkage and Selection via the Lasso.”
Journal of the Royal Statistical Society. Series B, Vol. 58, No. 1, 267–288.
Zou, H., and T. Hastie. (2005) “Regularization and Variable Selection via
the Elastic Net.” Journal of the Royal Statistical Society. Series B, Vol.
67, No. 2, 301–320.
Friedman, J., R. Tibshirani, and T. Hastie. (2010) “Regularization Paths

for Generalized Linear Models via Coordinate Descent.” Journal of
Statistical Software. Vol. 33, No. 1, https://www.jstatsoft.org/v33/i01
Hastie, T., R. Tibshirani, and J. Friedman. (2008) The Elements of

Statistical Learning. 2nd edition. New York: Springer.
THANK YOU

7SSMM700 Lecture 8

Uploaded by

Copyright:

Available Formats

7SSMM700 Lecture 8

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

7SSMM700 Lecture 8

Uploaded by

Copyright:

Available Formats

Regularization techniques and shrinkage estimators

7SSMM700 Quantitative Methods for Finance and Data Analytics

Leone Leonida and Marina Dolfin

King’s College London

The recent improvements in computer science are making data collection

Meanwhile, the analysis of large data sets poses methodological challenges

We explore the recent advancements in macroeconometrics and empirical

Elastic Net Regression

Variable selection in time series applications

CLRM vs. ML: an example in predicting returns on the Forex with

Multicollinearity and Overfitting

However, these assumptions tend to be false with sufficiently large

With less extreme situations multicollinearity can cause the model to be

Both LASSO and ridge regression estimation minimize the sum-of-squares

The penalty imposes a shrinkage on the coefficient estimates of ordinary

Generally, for the Lp regularization term we have

Ridge and LASSO deal with the L2 and L1 penalties respectively.

Regularization is used in preference over other common methods of

Ridge regression adds the L2 penalty such that we have

(y − X β)T (y − X β) + λβT β (2)

β̂ ridge = (X T X + λI )−1 X T y (3)

In equations (2) and (3), λ ≥ 0 is a tuning parameter for the penalty,

When λ = 0 we have the ordinary least squares estimate, and when

Ridge regression’s advantage over ordinary least squares lies in it’s

Ridge regression has a major disadvantage to other methods dealing with

Advantages The largest benefit of LASSO is the model’s ability to create

Disadvantages The main disadvantage with respect to Ridge is that

(y − X β)T (y − X β) such that βT β ≤ t (5)

(y − X β)T (y − X β) such that | β| ≤ t (6)

Figure: Image taken from 1

General penalty term

Model Linear regression model

Forecasting Forecasting future returns (60 minutes), using short-term

Tools Comparison CLRM vs. stepwise selection vs. ML by cross-validation

MACD Moving Average Convergence/Divergence

RSI Relative Strength Index

Future returns (60 mins in advance)

Look for the effectiveness of the prediction by adopting a back-testing

We calculate the returns by simply multiplying the actual returns by the

The Elastic Net Regularization is outperforming on the out-sample both

Friedman, J., R. Tibshirani, and T. Hastie. (2010) “Regularization Paths

Hastie, T., R. Tibshirani, and J. Friedman. (2008) The Elements of

You might also like