tensors-and-autograd

Duration: 8 min

This module delves into the fundamental building blocks of deep learning with PyTorch: tensors and autograd. Understanding tensors is crucial as they are the primary data structure for all computations in PyTorch. Autograd, on the other hand, is the automatic differentiation engine that powers neural network training. Mastery of these concepts is essential for effectively implementing and training deep learning models.

Visual: Computational Graph

        x (requires_grad=True)
        │
        ▼
    y = x * 2
        │
        ▼
    z = y + 3
        │
        ▼
    loss = z.sum()
        │
        ▼
    loss.backward()
        │
        ▼
    x.grad = ∂loss/∂x

Key Concepts Table

Understanding Tensors

Tensors in PyTorch are similar to NumPy's ndarrays, with the added capability of accelerating operations on a GPU. They can be scalars, vectors, matrices, or higher-dimensional arrays. Tensors can be created with various methods, including from data, random numbers, or existing data structures. They support a multitude of operations, which are crucial for building and training neural networks.

import torch

# Creating a tensor from a list
tensor_from_list = torch.tensor([1, 2, 3])
print('Tensor from list:', tensor_from_list)

# Creating a tensor with random numbers
random_tensor = torch.rand(3, 3)
print('Random tensor:', random_tensor)

# Creating a tensor from a NumPy array
import numpy as np
numpy_array = np.array([1, 2, 3])
tensor_from_numpy = torch.from_numpy(numpy_array)
print('Tensor from NumPy array:', tensor_from_numpy)

Try it in Google Colab:

Tensor from list: tensor([1, 2, 3])
Random tensor: 
tensor([[0.0236, 0.6493, 0.3440],
        [0.9439, 0.0332, 0.8675],
        [0.4316, 0.2699, 0.6487]])
Tensor from NumPy array: tensor([1, 2, 3])

Understanding Autograd

Autograd is PyTorch's automatic differentiation library that powers neural network training. It records operations performed on tensors and allows for the computation of gradients. This is essential for backpropagation, which is the method by which neural networks learn. Autograd works by building a computational graph on-the-fly, where nodes are tensors, and edges are functions.

import torch

# Create tensors
x = torch.tensor([1.0, 2.0, 3.0], requires_grad=True)
y = torch.tensor([4.0, 5.0, 6.0])

# Perform operations
z = x + y
out = z * z

# Compute gradients
out.backward(torch.tensor([1.0, 1.0, 1.0]))

# Print gradients
print('Gradient of x:', x.grad)

💡 Tip: Ensure that the 'requires_grad' attribute is set to True for tensors whose gradients you want to compute.

Practice Quizzes

Quiz 1: What does requires_grad=True do?

• Speeds up computation
• [✓] Enables gradient tracking
• Reduces memory
• Normalizes data

Quiz 2: What is a computational graph?

• A neural network
• [✓] History of operations for gradient computation
• A loss function
• A data structure

Quiz 3: When do you use .detach()?

• Training
• [✓] During inference/evaluation
• Computing loss
• Updating weights