Forest Hyperparameters

Node size: This sets the minimum number observations needed per leaf. It is actually a stopping criterion and it implicitly sets the depth of your trees.

• Smaller values create deeper trees.

• Larger values create smaller trees.

Number of features selected: At each split, only max_features (random) features are considered (instead of all \(p\)).

• Fewer features → greater tree diversity (better ensemble variance reduction) but weaker individual splits (higher bias).

• More features → stronger splits (lower bias) but higher correlation among trees (less ensemble variance reduction).

• Common defaults: \(\sqrt{p}\) for classification, \(p/3\) for regression.

Number of trees: This is exactly the number of bootstrap samples (one per tree).