Linear Regression Advanced Techniques

Duration: 5 min

This module delves into advanced techniques for Linear Regression, a fundamental supervised learning algorithm. We will explore methods to improve model performance, handle multicollinearity, and implement regularization techniques. Understanding these advanced techniques is crucial for building robust and accurate predictive models.

Handling Multicollinearity

Multicollinearity occurs when two or more predictor variables in a regression model are highly correlated, making it difficult to estimate the relationship between each predictor and the target variable. To handle multicollinearity, we can use techniques like Variance Inflation Factor (VIF) to identify and remove highly correlated features.

import pandas as pd
from statsmodels.stats.outliers_influence import variance_inflation_factor

# Sample data
data = {'A': [1, 2, 3, 4, 5], 'B': [2, 4, 6, 8, 10], 'C': [1, 0, 1, 0, 1]}
df = pd.DataFrame(data)

# Calculate VIF
X = df[['A', 'B', 'C']]
vif = pd.DataFrame()
vif["VIF"] = [variance_inflation_factor(X.values, i) for i in range(X.shape[1])]
vif["features"] = X.columns
print(vif)

Try it in Google Colab:

      VIF features
0  1.000000      A
1  6.333333      B
2  1.000000      C

Implementing Regularization

Regularization techniques like Ridge (L2) and Lasso (L1) regression help in reducing overfitting by adding a penalty to the loss function. Ridge regression adds the squared magnitude of coefficients as a penalty term, while Lasso adds the absolute value of magnitudes.

from sklearn.datasets import make_regression
from sklearn.linear_model import Ridge, Lasso
from sklearn.model_selection import train_test_split
import numpy as np

# Generate sample data
X, y = make_regression(n_samples=100, n_features=5, noise=0.1)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Ridge Regression
ridge = Ridge(alpha=1.0)
ridge.fit(X_train, y_train)
print('Ridge coefficients:', ridge.coef_)

# Lasso Regression
lasso = Lasso(alpha=0.1)
lasso.fit(X_train, y_train)
print('Lasso coefficients:', lasso.coef_)

💡 Tip: When using Lasso regression, be mindful that it can shrink some coefficients to zero, effectively performing feature selection. This can be useful for models with many features.

Key Concepts

Check Your Understanding