Gradient Boosting: Fundamentals

Duration: 5 min

This module delves into the fundamentals of Gradient Boosting, a powerful ensemble technique that combines weak learners to create a robust predictive model. Understanding Gradient Boosting is crucial for improving model accuracy and handling complex datasets efficiently.

Understanding Gradient Boosting

Gradient Boosting is an ensemble learning technique that builds models sequentially, where each new model aims to correct the errors of the previous one. It combines weak learners, typically decision trees, to create a strong learner. The key idea is to minimize the loss function by iteratively adding new models that focus on the residuals of the current ensemble.

import numpy as np
from sklearn.ensemble import GradientBoostingRegressor

# Sample data
X = np.array([[1, 2], [3, 4], [5, 6], [7, 8], [9, 10]])
y = np.array([3, 7, 11, 15, 19])

# Initialize and train the model
model = GradientBoostingRegressor(n_estimators=100, learning_rate=0.1, max_depth=1)
model.fit(X, y)

# Predict
predictions = model.predict(X)
print(predictions)

Try it in Google Colab:

[3.00000001 7.00000001 11. 15. 19.]

Key Parameters in Gradient Boosting

Several parameters influence the performance of Gradient Boosting models. n_estimators defines the number of boosting stages, learning_rate controls the contribution of each tree, and max_depth limits the complexity of individual trees. Tuning these parameters is essential for achieving optimal model performance.

import numpy as np
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split

# Generate a binary classification dataset
X, y = make_classification(n_samples=1000, n_features=20, n_classes=2, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Initialize and train the model
model = GradientBoostingClassifier(n_estimators=100, learning_rate=0.1, max_depth=3)
model.fit(X_train, y_train)

# Evaluate the model
accuracy = model.score(X_test, y_test)
print(f'Accuracy: {accuracy:.2f}')

💡 Tip: When tuning Gradient Boosting models, start with a higher number of estimators and a lower learning rate to ensure the model converges properly. Gradually increase the learning rate and decrease the number of estimators if the model overfits.