Hyperparameter Search

Grid Search: Form a Cartesian product of candidate values for each hyperparameter and exhaustively evaluate every combination. While comprehensive, the cost grows exponentially with the number of hyperparameters.

Random Search: Define a probability distribution for each hyperparameter and randomly sample combinations from the joint distribution over all hyperparameters. This explores the space more efficiently when only a few hyperparameters matter, but does not guarantee finding the global optimum.