building-neural-networks

Duration: 8 min

This module delves into the art and science of constructing neural networks using PyTorch, a powerful deep learning framework. We will explore the fundamental building blocks of neural networks, including layers, activation functions, and loss functions, and learn how to assemble them into cohesive models capable of tackling complex tasks.

Visual: Network Architecture

Input (784)
    │
    ▼
Hidden1 (128) ─ ReLU
    │
    ▼
Hidden2 (64) ─ ReLU
    │
    ▼
Output (10) ─ Softmax
    │
    ▼
Predictions

Key Concepts Table

Understanding Neural Network Layers

Neural network layers are the basic units of a neural network. Each layer performs a specific transformation on its input data. The most common types of layers include fully connected (dense) layers, convolutional layers, and recurrent layers. These layers are interconnected to form the network architecture, enabling the model to learn from data.

import torch
import torch.nn as nn

# Define a simple neural network with one hidden layer
class SimpleNN(nn.Module):
    def __init__(self):
        super(SimpleNN, self).__init__()
        # Input layer to hidden layer with ReLU activation
        self.hidden = nn.Linear(10, 5)
        # Hidden layer to output layer with ReLU activation
        self.predict = nn.Linear(5, 1)

    def forward(self, x):
        # Apply ReLU activation function
        x = torch.relu(self.hidden(x))
        # Output layer
        out = self.predict(x)
        return out

# Instantiate the model
model = SimpleNN()

Try it in Google Colab:

SimpleNN(
  (hidden): Linear(in_features=10, out_features=5, bias=True)
  (predict): Linear(in_features=5, out_features=1, bias=True)
)

Activation Functions

Activation functions introduce non-linearity into the model, allowing it to learn complex patterns. Common activation functions include ReLU (Rectified Linear Unit), Sigmoid, and Tanh. Each function has its own characteristics and is suited to different types of problems.

import torch
import torch.nn.functional as F

# Define a simple neural network with ReLU activation
class ActivationNN(nn.Module):
    def __init__(self):
        super(ActivationNN, self).__init__()
        self.fc1 = nn.Linear(10, 5)
        self.fc2 = nn.Linear(5, 1)

    def forward(self, x):
        # Apply ReLU activation function
        x = F.relu(self.fc1(x))
        # Apply Sigmoid activation function
        x = torch.sigmoid(self.fc2(x))
        return x

# Instantiate the model
model = ActivationNN()

💡 Tip: When using activation functions, ensure that the choice of function aligns with the nature of your problem. For example, ReLU is often preferred for hidden layers due to its simplicity and effectiveness in combating the vanishing gradient problem.

Practice Quizzes

Quiz 1: What does ReLU do?

• Normalizes data
• [✓] Applies max(0, x)
• Computes softmax
• Flattens tensors

Quiz 2: When do you use Softmax?

• Regression
• [✓] Multi-class classification
• Regression and classification
• Always

Quiz 3: What is the purpose of Dropout?

• Speed up training
• [✓] Reduce overfitting
• Normalize weights
• Compute gradients