Hyperparameter Tuning

Duration: 5 min

This module delves into the crucial process of hyperparameter tuning, a key step in optimizing machine learning models. Hyperparameters are settings that govern the training process and can significantly impact model performance. Understanding how to effectively tune these parameters is essential for achieving the best possible results from your machine learning algorithms.

Grid Search for Hyperparameter Tuning

Grid Search is a brute-force approach to hyperparameter tuning that systematically explores a predefined range of hyperparameter values. By evaluating the model's performance for each combination of parameters, Grid Search helps identify the optimal settings. This method is straightforward but can be computationally expensive, especially with a large parameter space.

from sklearn.datasets import load_iris
from sklearn.model_selection import GridSearchCV
from sklearn.svm import SVC

# Load dataset
iris = load_iris()
X, y = iris.data, iris.target

# Define parameter grid
param_grid = {'C': [0.1, 1, 10, 100], 'kernel': ['linear', 'rbf']}

# Initialize SVM classifier
svm = SVC()

# Perform Grid Search
grid_search = GridSearchCV(svm, param_grid, cv=5)
grid_search.fit(X, y)

# Print best parameters and score
print(f'Best parameters: {grid_search.best_params_}')
print(f'Best score: {grid_search.best_score_}')

Try it in Google Colab:

Best parameters: {'C': 1, 'kernel': 'linear'}
Best score: 0.98

Randomized Search for Hyperparameter Tuning

Randomized Search is an alternative to Grid Search that samples a fixed number of parameter combinations randomly from a specified distribution. This approach can be more efficient than Grid Search, especially when dealing with a large hyperparameter space, as it reduces the computational cost by exploring only a subset of the possible parameter values.

from sklearn.datasets import load_iris
from sklearn.model_selection import RandomizedSearchCV
from sklearn.svm import SVC
from scipy.stats import uniform

# Load dataset
iris = load_iris()
X, y = iris.data, iris.target

# Define parameter distributions
param_dist = {'C': uniform(loc=0, scale=4), 'kernel': ['linear', 'rbf']}

# Initialize SVM classifier
svm = SVC()

# Perform Randomized Search
random_search = RandomizedSearchCV(svm, param_distributions=param_dist, n_iter=100, cv=5)
random_search.fit(X, y)

# Print best parameters and score
print(f'Best parameters: {random_search.best_params_}')
print(f'Best score: {random_search.best_score_}')

💡 Tip: When using Grid Search, be mindful of the computational cost, especially with a large parameter space. Consider using Randomized Search as an alternative to reduce computation time.

Key Concepts

Check Your Understanding