AI HyperParameters (“cheatsheet”)

some hyperparameters are specific to certain models or algorithms.

Hyperparameters are crucial configurations set before training machine learning models, influencing their performance. Unlike model parameters learned during training, hyperparameters, such as learning rate, batch size, number of epochs, model architecture, activation functions, regularization techniques, and optimizer choice, must be manually set. Proper hyperparameter tuning can significantly enhance model performance, helping to avoid underfitting or overfitting. Techniques for tuning include manual adjustment, grid search, random search, Bayesian optimization, and Automated Hyperparameter Tuning (AutoML). The learning rate controls step size in optimization, batch size affects training speed and memory usage, and the architecture determines the model’s capacity. Activation functions introduce non-linearity, while regularization methods prevent overfitting. The choice of optimizer impacts how model weights are updated. Efficient hyperparameter tuning is essential for optimizing model accuracy and generalization to new data.

General Hyperparameters:

1. Learning Rate: Step size for weight updates.
2. Batch Size: Number of samples per gradient update.
3. Number of Epochs: Full cycles over the entire dataset.
4. Initialization Method: Strategy for initial weights (e.g., Xavier, He initialization).

Model Architecture (Deep Learning):

1. Number of Layers: Depth of the network.
2. Neurons per Layer: Width of the network or layers.
3. Activation Functions: ReLU, Sigmoid, Tanh, Leaky ReLU, etc.
4. Dropout Rate: Fraction of neurons to drop out for regularization.

Regularization:

1. L1 Regularization Coefficient: Regularization strength for L1.
2. L2 Regularization Coefficient: Regularization strength for L2.
3. Early Stopping: Stop training when a monitored metric stops improving.

Optimization:

1. Optimizer: SGD, Adam, RMSprop, etc.
2. Momentum: Parameter for accelerations in gradient descent.
3. Learning Rate Decay: Reduction method for the learning rate over time.

Convolutional Neural Networks (CNN):

1. Filter Size: Size of the convolutional kernels.
2. Stride: Stepsize for convolutional kernel movement.
3. Padding: Adding zeros to the input boundary for convolution.
4. Pooling Size: Dimension reduction factor through pooling operations.

Recurrent Neural Networks (RNN):

1. Sequence Length: Length of input sequences.
2. Hidden State Size: Dimensionality of RNN hidden states.
3. Cell Type: LSTM, GRU, or vanilla RNN.

Ensemble Methods (e.g., Random Forest, Gradient Boosting):

1. Number of Estimators: Number of trees in the forest or boosting rounds.
2. Max Depth: Maximum depth of each tree.
3. Min Samples Split: Minimum number of samples required to split a node.
4. Learning Rate (for boosting): Shrinks the contribution of each tree.

Support Vector Machines (SVM):

1. C (Regularization parameter): Trade-off between smooth decision boundary and classifying training points correctly.
2. Kernel Type: Linear, Polynomial, RBF, Sigmoid, etc.
3. Gamma: Kernel coefficient for ‘rbf’, ‘poly’, and ‘sigmoid’.

K-Nearest Neighbors (KNN):

1. Number of Neighbors: Number of nearest neighbors to consider for decisions.
2. Distance Metric: How distance is measured (e.g., Euclidean, Manhattan).

Clustering (e.g., K-Means):

1. Number of Clusters: The ‘K’ in K-Means.
2. Initialization Method: Method for selecting initial cluster centers.