NN Hyperparameters

What are some common neural network hyperparameters?

Answer

  • High-level hyperparameters:

    • Architecture type: For example MLP, CNN, LSTM, Transformer.

    • Number of hidden layers:

    • Number of neurons per layer:

    • Optimizer: The optimization algorithm used to minimize the loss function. Most used ones are Adam (Adaptive Moment Estimation), SGD or SGD with momentum.

    • Loss function: same idea as in classical ML. Depends on task.

    • Activation function: These functions introduce nonlinearity into the model.

  • Other hyperparameters:

    • Batch size: Number of observations used per parameter update (during training).

    • Number of epochs: How many times the model goes through the entire training dataset (during training).

    • Learning rate: Step size used when updating parameters.

    • Dropout rate: Percentage of neurons randomly dropped during training.

    • L1/L2 regularization: Penalties added to reduce overfitting and model complexity.

Back to collection