Out-of-Bag Error

What is out-of-bag (OOB) error in random forests, and how is it used for hyperparameter tuning?

Answer

When each tree is grown on a bootstrap sample, about \(36.8\%\) of the observations are left out (“out-of-bag”) for that tree. To compute OOB error:

  1. For each training observation, gather the predictions from all trees that did not include it in their bootstrap sample.

  2. Aggregate those predictions (average for regression, majority vote for classification) to form each observation’s OOB prediction. Note that an observation can be an used for an OOB prediction for different trees.

  3. Compare each OOB prediction to the true outcome and compute the overall error (e.g. mean squared error).

For hyperparameter tuning, you train forests with different settings (e.g. number of trees, max_features, min_samples_leaf) and choose the configuration that minimizes the OOB error. Finally, you retrain on the full training set with those hyperparameters and evaluate performance.

Back to collection