Advanced Topics in Unsupervised Learning
Duration: 45 min
Advanced Topics in Unsupervised Learning
Duration: 45 min
Introduction
Self-supervised learning leverages unlabeled data to create supervision signals, enabling models to learn rich representations without labels. Contrastive learning makes similar examples close and dissimilar ones distant. Generative models (GANs, normalizing flows) learn data distributions. Deep clustering combines representation learning with unsupervised clustering. These techniques power modern computer vision, NLP, and representation learning in domains without labels.
Self-Supervised Learning
Self-supervised learning creates labels from data itself. A common strategy: predicting masked/corrupted inputs, or distinguishing augmentations of the same sample from different samples.
import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.optim import AdamSelf-supervised learning: predict rotations of images
class RotationPredictor(nn.Module):
def __init__(self):
super().__init__()
self.conv1 = nn.Conv2d(1, 32, kernel_size=3, padding=1)
self.conv2 = nn.Conv2d(32, 64, kernel_size=3, padding=1)
self.pool = nn.MaxPool2d(2)
self.fc1 = nn.Linear(64 4 4, 128)
self.fc2 = nn.Linear(128, 4) # 4 rotations: 0°, 90°, 180°, 270°
def forward(self, x):
x = self.pool(F.relu(self.conv1(x)))
x = self.pool(F.relu(self.conv2(x)))
x = x.view(x.size(0), -1)
x = F.relu(self.fc1(x))
return self.fc2(x)Simulated data augmentation (rotations)
from torchvision.transforms import functional as TFdef rotate_batch(images, angles):
"""Rotate images by angles (in degrees)"""
rotated = []
for img, angle in zip(images, angles):
# img shape: (1, 8, 8)
img_pil = TF.to_pil_image(img.cpu().numpy()[0])
rotated_pil = TF.rotate(img_pil, float(angle))
rotated_tensor = torch.FloatTensor(np.array(rotated_pil)).unsqueeze(0)
rotated.append(rotated_tensor)
return torch.stack(rotated)
Load digits
from sklearn.datasets import load_digits
digits = load_digits()
X = torch.FloatTensor(digits.data[:200].reshape(-1, 1, 8, 8)) / 255Generate rotations
rotations = [0, 90, 180, 270]
rotation_labels = [0, 1, 2, 3]print("Self-supervised rotation prediction:")
print(f"Input shape: {X.shape}")
Train to predict rotation
model = RotationPredictor()
optimizer = Adam(model.parameters(), lr=0.01)
criterion = nn.CrossEntropyLoss()for epoch in range(5):
for rotation_label, angle in zip(rotation_labels, rotations):
rotated_x = torch.rot90(X, k=rotation_label // 2)
logits = model(rotated_x)
target = torch.full((rotated_x.size(0),), rotation_label, dtype=torch.long)
loss = criterion(logits, target)
optimizer.zero_grad()
loss.backward()
optimizer.step()
if (epoch + 1) % 2 == 0:
with torch.no_grad():
pred = model(X)
acc = (pred.argmax(1) == 0).float().mean()
print(f" Epoch {epoch + 1}: Loss={loss.item():.3f}, Acc on 0°={acc:.1%}")
Contrastive Learning: SimCLR
Contrastive methods learn by pulling augmented versions of the same image close while pushing different images apart.
import torch
import torch.nn as nn
import torch.nn.functional as Fclass SimCLRProjector(nn.Module):
def __init__(self, in_dim=128, hidden_dim=256, out_dim=128):
super().__init__()
self.fc1 = nn.Linear(in_dim, hidden_dim)
self.fc2 = nn.Linear(hidden_dim, out_dim)
def forward(self, h):
h = F.relu(self.fc1(h))
return self.fc2(h)
Simulated encoder + projector
class SimCLRModel(nn.Module):
def __init__(self, encoder_dim=64, proj_dim=128):
super().__init__()
# Encoder (simplified)
self.encoder = nn.Sequential(
nn.Linear(28*28, 128),
nn.ReLU(),
nn.Linear(128, encoder_dim)
)
self.projector = SimCLRProjector(encoder_dim, hidden_dim=256, out_dim=proj_dim)
def forward(self, x):
h = self.encoder(x)
z = self.projector(h)
return h, zdef contrastive_loss(z_i, z_j, temperature=0.07, batch_size=32):
"""NT-Xent loss (normalized temperature-scaled cross entropy)"""
z = torch.cat([z_i, z_j], dim=0) # 2B samples
# Similarity matrix
sim_matrix = F.cosine_similarity(z.unsqueeze(1), z.unsqueeze(0), dim=2)
# Positives: diagonal blocks
pos_mask = torch.eye(batch_size * 2, dtype=torch.bool).to(z.device)
pos_mask = pos_mask | torch.eye(batch_size * 2, k=batch_size, dtype=torch.bool).to(z.device)
pos_mask = pos_mask | torch.eye(batch_size * 2, k=-batch_size, dtype=torch.bool).to(z.device)
# Negatives: all others
neg_mask = ~pos_mask
# Scale by temperature
sim_matrix = sim_matrix / temperature
# NT-Xent
loss = -torch.log(
torch.exp(sim_matrix[pos_mask]).sum(dim=1) /
torch.exp(sim_matrix[neg_mask]).sum(dim=1)
)
return loss.mean()
Demonstration
batch_size = 8
model = SimCLRModel(encoder_dim=64, proj_dim=128)Dummy augmented pairs
x_base = torch.randn(batch_size, 28*28)
h_i, z_i = model(x_base)
h_j, z_j = model(x_base + 0.1 * torch.randn_like(x_base)) # Augmented versionloss = contrastive_loss(z_i, z_j, batch_size=batch_size)
print(f"Contrastive loss: {loss.item():.3f}")
print(f"Encoder output shape: {h_i.shape}")
Generative Adversarial Networks (GANs)
GANs learn data distribution by competition: a generator creates fake samples; a discriminator distinguishes fake from real. This drives the generator to learn the real distribution.
import torch
import torch.nn as nn
from torch.optim import Adamclass Generator(nn.Module):
def __init__(self, latent_dim=20, image_dim=64):
super().__init__()
self.fc1 = nn.Linear(latent_dim, 128)
self.fc2 = nn.Linear(128, 256)
self.fc3 = nn.Linear(256, image_dim)
def forward(self, z):
h = torch.relu(self.fc1(z))
h = torch.relu(self.fc2(h))
x = torch.tanh(self.fc3(h))
return x
class Discriminator(nn.Module):
def __init__(self, image_dim=64):
super().__init__()
self.fc1 = nn.Linear(image_dim, 128)
self.fc2 = nn.Linear(128, 64)
self.fc3 = nn.Linear(64, 1)
def forward(self, x):
h = torch.relu(self.fc1(x))
h = torch.relu(self.fc2(h))
return torch.sigmoid(self.fc3(h))
Initialize
latent_dim = 20
image_dim = 64
generator = Generator(latent_dim, image_dim)
discriminator = Discriminator(image_dim)gen_opt = Adam(generator.parameters(), lr=0.0002, betas=(0.5, 0.999))
disc_opt = Adam(discriminator.parameters(), lr=0.0002, betas=(0.5, 0.999))
criterion = nn.BCELoss()
Training loop (simplified)
print("GAN training:")
for epoch in range(3):
# Real data
real_data = torch.randn(32, image_dim) * 2 # Simulate real distribution
# Discriminator: maximize log(D(x)) + log(1-D(G(z)))
disc_opt.zero_grad()
# Real samples
real_pred = discriminator(real_data)
real_loss = criterion(real_pred, torch.ones_like(real_pred))
# Fake samples
z = torch.randn(32, latent_dim)
fake_data = generator(z)
fake_pred = discriminator(fake_data.detach())
fake_loss = criterion(fake_pred, torch.zeros_like(fake_pred))
disc_loss = real_loss + fake_loss
disc_loss.backward()
disc_opt.step()
# Generator: maximize log(D(G(z)))
gen_opt.zero_grad()
fake_pred = discriminator(fake_data)
gen_loss = criterion(fake_pred, torch.ones_like(fake_pred))
gen_loss.backward()
gen_opt.step()
if (epoch + 1) % 1 == 0:
print(f" Epoch {epoch + 1}: D_loss={disc_loss.item():.3f}, G_loss={gen_loss.item():.3f}")Generate samples
with torch.no_grad():
z_sample = torch.randn(5, latent_dim)
generated = generator(z_sample)
print(f"Generated samples shape: {generated.shape}")
Normalizing Flows
Normalizing flows learn invertible transformations mapping data to a simple distribution (e.g., Gaussian). Unlike autoencoders, they provide exact likelihood and can generate samples.
import torch
import torch.nn as nn
from torch.distributions import Normal, TransformedDistribution
from torch.distributions.transforms import ComposeTransform, AffineTransformSimple flow: affine transformation
class SimpleAffineFlow(nn.Module):
def __init__(self, input_dim):
super().__init__()
self.scale = nn.Parameter(torch.randn(input_dim))
self.shift = nn.Parameter(torch.randn(input_dim))
def forward(self, x):
z = (x - self.shift) / (torch.exp(self.scale) + 1e-6)
log_det = -self.scale.sum()
return z, log_det
def inverse(self, z):
x = z * torch.exp(self.scale) + self.shift
return xTrain on 2D Gaussian mixture
model = SimpleAffineFlow(2)
optimizer = Adam(model.parameters(), lr=0.01)Real data: mixture of Gaussians
def sample_data(n=128):
mean1 = torch.tensor([2.0, 2.0])
mean2 = torch.tensor([-2.0, -2.0])
samples = []
for _ in range(n // 2):
samples.append(torch.randn(1, 2) + mean1)
samples.append(torch.randn(1, 2) + mean2)
return torch.cat(samples)print("Normalizing Flow training:")
for epoch in range(5):
data = sample_data(32)
# Forward pass
z, log_det = model(data)
# Likelihood under standard normal
base_dist = Normal(torch.zeros(2), torch.ones(2))
log_prob_z = base_dist.log_prob(z).sum(dim=1)
# Total log likelihood
log_prob_x = log_prob_z + log_det
loss = -log_prob_x.mean()
optimizer.zero_grad()
loss.backward()
optimizer.step()
if (epoch + 1) % 2 == 0:
print(f" Epoch {epoch + 1}: Loss={loss.item():.3f}")
Generate samples by inverting standard normal
with torch.no_grad():
z_samples = torch.randn(10, 2)
x_samples = model.inverse(z_samples)
print(f"Generated samples shape: {x_samples.shape}")
Deep Clustering
Combine representation learning with clustering. Learn embeddings where similar items cluster tightly.
import torch
import torch.nn as nn
from sklearn.cluster import KMeansclass DeepClusteringModel(nn.Module):
def __init__(self, input_dim, embedding_dim=64, n_clusters=10):
super().__init__()
self.encoder = nn.Sequential(
nn.Linear(input_dim, 128),
nn.ReLU(),
nn.Linear(128, embedding_dim)
)
self.cluster_head = nn.Linear(embedding_dim, n_clusters)
self.n_clusters = n_clusters
def forward(self, x):
embeddings = self.encoder(x)
logits = self.cluster_head(embeddings)
return embeddings, logits
Generate synthetic data
np.random.seed(42)
X1 = np.random.randn(50, 10) + np.array([5, 5, 0, 0, 0, 0, 0, 0, 0, 0])
X2 = np.random.randn(50, 10) + np.array([-5, -5, 0, 0, 0, 0, 0, 0, 0, 0])
X3 = np.random.randn(50, 10) + np.array([0, 0, 5, 5, 0, 0, 0, 0, 0, 0])X = np.vstack([X1, X2, X3])
X_tensor = torch.FloatTensor(X)
model = DeepClusteringModel(input_dim=10, embedding_dim=32, n_clusters=3)
optimizer = Adam(model.parameters(), lr=0.01)
Deep clustering loop
print("Deep Clustering training:")
for epoch in range(10):
# Get embeddings
embeddings, logits = model(X_tensor)
# Cluster embeddings with KMeans
embeddings_np = embeddings.detach().numpy()
kmeans = KMeans(n_clusters=3, random_state=42, n_init=5)
cluster_labels = kmeans.fit_predict(embeddings_np)
# Loss: pull embeddings toward cluster centers
cluster_loss = 0
for k in range(3):
mask = cluster_labels == k
if mask.sum() > 0:
center = embeddings[mask].mean(dim=0)
distances = torch.norm(embeddings[mask] - center, dim=1)
cluster_loss += distances.mean()
optimizer.zero_grad()
cluster_loss.backward()
optimizer.step()
if (epoch + 1) % 3 == 0:
print(f" Epoch {epoch + 1}: Clustering loss={cluster_loss.item():.3f}")Final clustering
with torch.no_grad():
embeddings_final, _ = model(X_tensor)
final_clusters = KMeans(n_clusters=3, random_state=42, n_init=5).fit_predict(embeddings_final)
print(f"Final cluster assignments: {final_clusters[:5]}...")
Key Takeaways
- Self-supervised learning: Create supervision from data itself (rotations, masking, augmentations)
- Contrastive learning: Pull similar examples close; push different ones apart
- GANs: Generator + Discriminator competition learns realistic distributions
- Normalizing flows: Invertible transformations; exact likelihood; generate samples
- Deep clustering: Learn representations and cluster simultaneously
- These methods enable learning from massive unlabeled datasets
---
Quiz
Question 1: Self-supervised learning creates labels by:
- A) Requiring human annotation
- B) Using the data itself (rotations, masking, augmentations) (✓)
- C) Random assignment
- D) Copying from labeled datasets
Question 2: In contrastive learning (SimCLR), what is the goal?
- A) Classify images into categories
- B) Make augmentations of same image similar and different images distant (✓)
- C) Generate new images
- D) Compress data to lower dimension
Question 3: In a GAN, the Generator is trained to:
- A) Classify real vs fake samples
- B) Create samples that fool the Discriminator (✓)
- C) Compress data
- D) Cluster unlabeled data
Question 4: Normalizing Flows provide which advantage over autoencoders?
- A) Faster training
- B) Exact likelihood and invertibility (✓)
- C) Easier to implement
- D) Smaller model size
Question 5: Deep clustering combines:
- A) Supervised classification with regression
- B) Multiple GANs in ensemble
- C) Representation learning with unsupervised clustering (✓)
- D) Only K-means with no neural network
Question 6: Self-supervised rotation prediction works because:
- A) Rotation angle is always predictable from pixels
- B) The model learns useful image features to solve the pretext task (✓)
- C) Rotation preserves all information
- D) It requires labels of true rotation angles