Boosting with MSE

What are the general steps of the gradient boosting algorithm when using the mean squared error loss function $L(y, F(x)) = \tfrac{1}{2}(y - F(x))^2$ ?

Answer

Inputs: Training data $\{(x_i, y_i)\}_{i=1}^n$ , loss function

L(y, F(x)) = \frac{1}{2}\bigl(y - F(x)\bigr)^2,

number of boosting iterations $$M$$ , learning rate $\eta \in (0,1]$ .

Initialization (initial weak learner):
$F_{0}(x) = \arg\min_{\gamma}\sum_{i=1}^{n}\frac{1}{2}(y_{i} - \gamma)^2 = \frac{1}{n}\sum_{i=1}^n y_i.$
Thus $F_{0}(x)$ is taken as the mean of the targets.
For $$m=1$$ to $$M$$ :
1. Compute pseudo‐residuals: For each $$i$$ ,
  $r_{i}^{(m)} = -\left.\frac{\partial}{\partial F(x_i)}\frac{1}{2}\bigl(y_i - F(x_i)\bigr)^2\right|_{F=F_{m-1}} = y_i - F_{m-1}(x_i).$
  Here as we see, the pseudo-residuals are the "normal" residuals.
2. Fit a weak learner: Fit a weak learner (typically a decision tree) to the training set $\{(x_i, r_i^{(m)})\}_{i=1}^n$ , using each pseudo‐residual $r_i^{(m)}$ as the target.
3. Update the model:
  $F_{m}(x) = F_{m-1}(x) + \eta \, h_{m}(x),$
  where $$h_m(x)$$ is the new weak learner’s prediction of the pseudo‐residuals, scaled by the learning rate $\eta$ .
Final model:
$F_{M}(x)$