Functional Gradient Descent
Why is it called “gradient” boosting?
Answer
Gradient boosting earns its name because at each step it fits a new weak learner to the negative gradient (pseudo‐residuals) of the loss function w.r.t. the predictions of the current model:
\[
r_i^{(m)}
= -\frac{\partial L\bigl(y_i, F_{m-1}(x_i)\bigr)}{\partial F_{m-1}(x_i)}.
\]
Adding this learner (scaled by the learning rate) mimics a gradient‐descent but in prediction space (function space; not parameter space), hence “gradient” boosting.