Practical Applications of Ensemble Learning

Duration: 7 min

This module delves into the practical applications of ensemble learning techniques such as Bagging, Boosting, XGBoost, LightGBM, CatBoost, Stacking, and Voting. Understanding these methods is crucial for improving model performance and achieving better results in machine learning projects.

Bagging: Averaging Multiple Models

Bagging, or Bootstrap Aggregating, involves training multiple models on different subsets of the training data and then averaging their predictions. This technique reduces variance and helps to avoid overfitting. Common algorithms that use bagging include Random Forests.

from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split

# Generate a random binary classification dataset
X, y = make_classification(n_samples=1000, n_features=20, n_classes=2, random_state=42)

# Split the dataset into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Initialize the RandomForestClassifier
rf = RandomForestClassifier(n_estimators=100, random_state=42)

# Train the model
rf.fit(X_train, y_train)

# Make predictions
y_pred = rf.predict(X_test)

# Print the accuracy
print(f'Accuracy: {rf.score(X_test, y_test):.2f}')

Try it in Google Colab:

Accuracy: 0.85

Boosting: Sequentially Training Models

Boosting is an ensemble technique where multiple models are trained sequentially. Each model attempts to correct the errors of the previous one. Popular boosting algorithms include AdaBoost, XGBoost, LightGBM, and CatBoost.

import xgboost as xgb
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split

# Generate a random binary classification dataset
X, y = make_classification(n_samples=1000, n_features=20, n_classes=2, random_state=42)

# Split the dataset into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Initialize the XGBoost classifier
xgb_clf = xgb.XGBClassifier(n_estimators=100, learning_rate=0.1, random_state=42)

# Train the model
xgb_clf.fit(X_train, y_train)

# Make predictions
y_pred = xgb_clf.predict(X_test)

# Print the accuracy
print(f'Accuracy: {xgb_clf.score(X_test, y_test):.2f}')

💡 Tip: When using boosting algorithms like XGBoost, tuning hyperparameters such as learning_rate and n_estimators can significantly impact model performance.