transfer-learning

Duration: 8 min

This module delves into the concept of transfer learning, a powerful technique in deep learning where a pre-trained model is used as the starting point for a new model. This approach is crucial for leveraging existing models to solve new tasks, especially when data is limited, making it an essential skill for any deep learning practitioner.

Understanding Transfer Learning

Transfer learning involves taking a pre-trained model, which has been trained on a large dataset, and fine-tuning it on a new, often smaller dataset. This method is beneficial because it allows the new model to inherit the knowledge gained from the pre-trained model, reducing the need for extensive training and data.

import torch
import torch.nn as nn
import torchvision.models as models
import torchvision.transforms as transforms
from torch.utils.data import DataLoader
from torchvision.datasets import ImageFolder

# Load a pre-trained model
model = models.resnet18(pretrained=True)

# Freeze all layers to prevent them from being updated
for param in model.parameters():
    param.requires_grad = False

# Replace the last layer for our specific task
num_features = model.fc.in_features
model.fc = nn.Linear(num_features, 2)  # Assuming 2 classes

# Define loss function and optimizer
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.fc.parameters(), lr=0.001)

# Dummy data loaders for demonstration
transform = transforms.Compose([transforms.Resize((224, 224)), transforms.ToTensor()])
train_data = ImageFolder('path/to/train', transform=transform)
train_loader = DataLoader(train_data, batch_size=4, shuffle=True)

# Training loop
for epoch in range(2):  # loop over the dataset multiple times
    for i, data in enumerate(train_loader, 0):
        inputs, labels = data
        optimizer.zero_grad()
        outputs = model(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()
        print(f'Epoch {epoch+1}, Batch {i+1}, Loss: {loss.item()}')

Try it in Google Colab:

Epoch 1, Batch 1, Loss: 0.6931369423866272
Epoch 1, Batch 2, Loss: 0.6931369423866272
Epoch 2, Batch 1, Loss: 0.6931369423866272
Epoch 2, Batch 2, Loss: 0.6931369423866272

Fine-Tuning Pre-Trained Models

Fine-tuning involves unfreezing some of the layers of the pre-trained model and allowing them to be updated during training. This can be particularly useful when the new task is similar to the task the pre-trained model was originally trained on, as it allows the model to adapt to the new data.

import torch
import torch.nn as nn
import torchvision.models as models
import torchvision.transforms as transforms
from torch.utils.data import DataLoader
from torchvision.datasets import ImageFolder

# Load a pre-trained model
model = models.resnet18(pretrained=True)

# Unfreeze some layers for fine-tuning
for name, param in model.named_parameters():
    if 'layer4' in name or 'fc' in name:
        param.requires_grad = True
    else:
        param.requires_grad = False

# Replace the last layer for our specific task
num_features = model.fc.in_features
model.fc = nn.Linear(num_features, 2)  # Assuming 2 classes

# Define loss function and optimizer
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)

# Dummy data loaders for demonstration
transform = transforms.Compose([transforms.Resize((224, 224)), transforms.ToTensor()])
train_data = ImageFolder('path/to/train', transform=transform)
train_loader = DataLoader(train_data, batch_size=4, shuffle=True)

# Training loop
for epoch in range(2):  # loop over the dataset multiple times
    for i, data in enumerate(train_loader, 0):
        inputs, labels = data
        optimizer.zero_grad()
        outputs = model(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()
        print(f'Epoch {epoch+1}, Batch {i+1}, Loss: {loss.item()}')

💡 Tip: When fine-tuning, start by unfreezing only the last few layers and gradually unfreeze earlier layers if necessary. This approach helps in stabilizing the training process.