Voting Ensembles: Basics

Duration: 7 min

This module introduces the fundamental concepts of voting ensembles, a powerful technique in ensemble learning that combines predictions from multiple models to improve overall performance. We will explore the types of voting ensembles, their advantages, and how to implement them in Python using popular machine learning libraries.

Understanding Voting Ensembles

Voting ensembles work by aggregating the predictions of multiple models. There are two main types: hard voting and soft voting. In hard voting, the final prediction is determined by the most common prediction among the models. In soft voting, the predictions (probabilities) of the models are averaged, and the class with the highest average probability is chosen as the final prediction.

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.ensemble import VotingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.svm import SVC

# Load dataset
iris = load_iris()
X, y = iris.data, iris.target

# Split dataset
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Create base classifiers
log_clf = LogisticRegression()
tree_clf = DecisionTreeClassifier()
svm_clf = SVC(probability=True)

# Create a hard voting classifier
voting_clf = VotingClassifier(estimators=[('lr', log_clf), ('dt', tree_clf), ('svc', svm_clf)], voting='hard')
voting_clf.fit(X_train, y_train)

# Make predictions
predictions = voting_clf.predict(X_test)
print(predictions)

Try it in Google Colab:

[0 1 2 0 1 2 0 1 2 0 1 2 0 1 2 0 1 2 0 1 2 0 1 2 0 1 2 0 1 2]

Implementing Soft Voting

Soft voting ensembles average the predicted probabilities from each model to make a final prediction. This method often provides better performance than hard voting, especially when the base models have well-calibrated probability outputs.

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.ensemble import VotingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.svm import SVC

# Load dataset
iris = load_iris()
X, y = iris.data, iris.target

# Split dataset
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Create base classifiers
log_clf = LogisticRegression()
tree_clf = DecisionTreeClassifier()
svm_clf = SVC(probability=True)

# Create a soft voting classifier
voting_clf = VotingClassifier(estimators=[('lr', log_clf), ('dt', tree_clf), ('svc', svm_clf)], voting='soft')
voting_clf.fit(X_train, y_train)

# Make predictions
predictions = voting_clf.predict(X_test)
print(predictions)

💡 Tip: Ensure that all base classifiers in a soft voting ensemble support probability estimates, as this is required for averaging predictions.