Logistic Regression Advanced Techniques

Duration: 5 min

This module delves into advanced techniques for optimizing and implementing logistic regression models. We will explore regularization methods, multiclass classification, and handling imbalanced datasets. Understanding these techniques is crucial for improving model performance and making accurate predictions in complex scenarios.

Regularization in Logistic Regression

Regularization is a technique used to prevent overfitting by adding a penalty term to the loss function. In logistic regression, L1 (Lasso) and L2 (Ridge) regularization are commonly used. L1 regularization adds the absolute value of the coefficients, promoting sparsity, while L2 regularization adds the squared value of the coefficients, encouraging smaller coefficients.

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split

# Generate a synthetic dataset
X, y = make_classification(n_samples=100, n_features=20, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Apply L2 regularization
log_reg = LogisticRegression(penalty='l2', C=1.0, solver='lbfgs', random_state=42)
log_reg.fit(X_train, y_train)

# Predict and evaluate
y_pred = log_reg.predict(X_test)
print('Accuracy:', log_reg.score(X_test, y_test))

Try it in Google Colab:

Accuracy: 0.85

Handling Imbalanced Datasets

Imbalanced datasets, where one class significantly outnumbers the other, can lead to biased models. Techniques like class weighting and over/under-sampling can be used to address this issue. Class weighting assigns higher penalties to misclassifications of the minority class, while over/under-sampling adjusts the dataset to balance the classes.

from imblearn.over_sampling import SMOTE
from sklearn.metrics import classification_report

# Apply SMOTE to balance the dataset
smote = SMOTE(random_state=42)
X_resampled, y_resampled = smote.fit_resample(X_train, y_train)

# Train logistic regression on the resampled dataset
log_reg_balanced = LogisticRegression(solver='lbfgs', random_state=42)
log_reg_balanced.fit(X_resampled, y_resampled)

# Predict and evaluate
y_pred_balanced = log_reg_balanced.predict(X_test)
print(classification_report(y_test, y_pred_balanced))

💡 Tip: When dealing with imbalanced datasets, always evaluate model performance using metrics like precision, recall, and F1-score, in addition to accuracy, to get a comprehensive understanding of model performance.

Key Concepts

Check Your Understanding