Building Neural Networks

Duration: 8 min

This module delves into the intricacies of constructing neural networks using Python, a fundamental skill for AI development. We will explore the architecture of neural networks, the process of initializing weights, and the mechanics of forward and backward propagation. Understanding these concepts is crucial for developing effective machine learning models.

Understanding Neural Network Architecture

A neural network consists of layers of interconnected nodes, or neurons, which process and transmit information. The architecture typically includes an input layer, one or more hidden layers, and an output layer. Each neuron receives input, applies a weight to it, sums these inputs, and passes the result through an activation function to produce an output.

example1.py

import numpy as np

# Initialize weights and biases for a simple neural network
weights = {
    'node0': np.array([0.5, 0.5]),
    'node1': np.array([0.3, 0.3])
}
biases = {
    'node0': 0.1,
    'node1': -0.1
}

# Define the activation function (sigmoid)
def sigmoid(x):
    return 1 / (1 + np.exp(-x))

# Forward propagation function
def forward_propagation(inputs, weights, biases):
    hidden_layer = sigmoid(np.dot(inputs, weights['node0']) + biases['node0'])
    output_layer = sigmoid(np.dot(hidden_layer, weights['node1']) + biases['node1'])
    return hidden_layer, output_layer

# Example input
inputs = np.array([0.1, 0.2])

# Perform forward propagation
hidden_layer, output_layer = forward_propagation(inputs, weights, biases)
print('Hidden Layer Output:', hidden_layer)
print('Output Layer Output:', output_layer)

Try it in Google Colab:

Hidden Layer Output: [0.5986976]
Output Layer Output: [0.57672185]

Implementing Backpropagation

Backpropagation is a method used to update the weights and biases of a neural network based on the error between the predicted output and the actual output. It involves calculating the gradient of the loss function with respect to each weight and bias, and then adjusting them in the opposite direction of the gradient to minimize the loss.

example2.py

import numpy as np

# Define the loss function (Mean Squared Error)
def mse_loss(y_true, y_pred):
    return np.mean((y_true - y_pred) ** 2)

# Define the derivative of the loss function
def mse_loss_derivative(y_true, y_pred):
    return 2 * (y_pred - y_true) / y_true.size

# Backpropagation function
def backpropagation(inputs, weights, biases, outputs, y_true, learning_rate):
    # Forward propagation
    hidden_layer, output_layer = forward_propagation(inputs, weights, biases)

    # Calculate the loss
    loss = mse_loss(y_true, output_layer)

    # Calculate the gradient of the loss with respect to the output layer
    d_loss_output = mse_loss_derivative(y_true, output_layer)

    # Calculate the gradient of the loss with respect to the hidden layer
    d_loss_hidden = d_loss_output * hidden_layer * (1 - hidden_layer)

    # Update the weights and biases
    weights['node1'] -= learning_rate * np.dot(np.atleast_2d(hidden_layer), np.atleast_2d(d_loss_output))
    biases['node1'] -= learning_rate * d_loss_output
    weights['node0'] -= learning_rate * np.dot(np.atleast_2d(inputs), np.atleast_2d(d_loss_hidden))
    biases['node0'] -= learning_rate * d_loss_hidden

    return loss

# Example input and target output
inputs = np.array([0.1, 0.2])
y_true = np.array([0.8])
learning_rate = 0.1

# Perform backpropagation
loss = backpropagation(inputs, weights, biases, outputs, y_true, learning_rate)
print('Loss:', loss)

💡 Tip: Ensure that your learning rate is set appropriately; a rate that is too high can cause the model to diverge, while a rate that is too low can result in very slow convergence.