Ridge Regression
What is ridge regression?
Answer
Ridge regression is a form of penalised (linear) regression in which the estimation of the coefficients is done by minimizing the sum of squared residuals (OLS) plus an \(\ell_2\) penalty on the coefficients:
\[
L_{\rm ridge}(\beta)
= \sum_{i=1}^n (Y_i - x_i^\top\beta)^2
+ \lambda \sum_{j=1}^p \beta_j^2.
\]
Here \(\lambda\) is called the regularization parameter (also known as the shrinkage parameter), and it is a hyperparameter of this regression model.
The solution has the closed‐form
\[
\hat\beta_{\rm ridge}
= \bigl(X^\top X + \lambda I\bigr)^{-1} X^\top Y,
\]
The estimated slope coefficients from ridge regression are smaller in an \(\ell_2\) sense than those from OLS, i.e.
\[
\sqrt{\sum_{j=1}^p \hat\beta_{\mathrm{ridge},j}^2}
\;\le\;
\sqrt{\sum_{j=1}^p \hat\beta_{\mathrm{OLS},j}^2}.
\]
Notes and comments
Note: The \(\ell_2\) norm \(\|\beta\|_2 = \sqrt{\sum_{j=1}^p \beta_j^2}\) is the Euclidean distance of the vector \(\beta\) from the origin (this comes from the Pythagorean Theorem).