Information Theory Basics

Duration: 5 min

This module introduces the fundamental concepts of Information Theory, a crucial area in machine learning that deals with the quantification, storage, and communication of information. Understanding these basics is essential for optimizing machine learning models and algorithms, as it helps in assessing the efficiency and effectiveness of data processing and model performance.

Entropy

Entropy is a measure of the uncertainty or randomness of a random variable. In information theory, it quantifies the amount of information produced by a stochastic source of data. Higher entropy indicates greater uncertainty. For a discrete random variable X with possible outcomes {x1, x2,..., xn} and corresponding probabilities {p1, p2,..., pn}, the entropy H(X) is given by H(X) = -sum(pi * log2(pi)) for i in 1 to n.

import math

# Define probabilities
probabilities = [0.1, 0.2, 0.3, 0.4]

# Calculate entropy
entropy = -sum(p * math.log2(p) for p in probabilities if p > 0)
print(f'Entropy: {entropy}')

Try it in Google Colab:

Entropy: 1.8464393895386665

Kullback-Leibler Divergence

Kullback-Leibler (KL) Divergence measures how one probability distribution diverges from a second, expected probability distribution. It is a non-symmetric measure of the difference between two probability distributions P and Q. KL Divergence is particularly useful in machine learning for comparing the predicted distribution with the true distribution.

import math

# Define two probability distributions
P = [0.2, 0.3, 0.5]
Q = [0.3, 0.4, 0.3]

# Calculate KL Divergence
kl_divergence = sum(p * math.log2(p/q) for p, q in zip(P, Q) if p > 0 and q > 0)
print(f'KL Divergence: {kl_divergence}')

💡 Tip: When calculating KL Divergence, ensure that both distributions P and Q are properly normalized and that none of the probabilities are zero to avoid undefined or infinite results.

Key Concepts

Check Your Understanding