Introduction to Bayesian Inference

Duration: 45 min

Introduction to Bayesian Inference

Duration: 45 min

Bayesian inference represents a fundamentally different approach to statistical reasoning than the frequentist framework. Rather than viewing parameters as fixed unknown constants, Bayesian inference treats parameters as random variables with probability distributions. This perspective aligns naturally with how we update beliefs based on evidence—a core aspect of human reasoning and machine learning.

The Bayesian Framework

The Bayesian approach centers on three key components: the prior, the likelihood, and the posterior.

  • Prior: Your belief about a parameter before observing data, denoted p(θ)
  • Likelihood: The probability of observing the data given parameter values, p(D|θ)
  • Posterior: Your updated belief after observing data, p(θ|D)

These connect through Bayes' Rule:

p(θ|D) = p(D|θ) × p(θ) / p(D)

Where p(D) = ∫ p(D|θ)p(θ)dθ is the marginal likelihood or evidence.

Bayes' Rule Intuition

Bayes' rule tells us how to rationally update beliefs when new evidence arrives:

import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import binom, norm
Simple coin flip example


Prior: belief about probability of heads

prior_alpha = 1  # Beta(1, 1) is uniform
prior_beta = 1
Data: 7 heads out of 10 flips

heads = 7
tails = 3
Likelihood for this data depends on p (prob of heads)


We'll plot likelihood function
p_vals = np.linspace(0, 1, 100)
likelihood = binom.pmf(heads, heads + tails, p_vals)
likelihood = likelihood / likelihood.max()  # Normalize for display
Prior (uniform)

prior = np.ones_like(p_vals)
prior = prior / prior.sum()
Posterior (proportional to prior × likelihood)

posterior_unnormalized = prior * likelihood
posterior = posterior_unnormalized / posterior_unnormalized.sum()
plt.figure(figsize=(10, 6))
plt.plot(p_vals, prior, label='Prior: Uniform', linewidth=2)
plt.plot(p_vals, likelihood, label='Likelihood: 7 heads/10 flips', linewidth=2)
plt.plot(p_vals, posterior, label='Posterior', linewidth=2.5, color='red')
plt.xlabel('Probability of Heads (p)')
plt.ylabel('Density')
plt.legend()
plt.title("Bayes' Rule: Updating Belief from Data")
plt.grid(alpha=0.3)
plt.show()
Posterior mean

posterior_mean = np.sum(p_vals * posterior / posterior.sum())
print(f"Posterior mean (best estimate of p): {posterior_mean:.3f}")

Conjugate Priors

A conjugate prior is one where the posterior has the same functional form as the prior. This mathematical convenience enables closed-form solutions instead of numerical computation.

Beta-Binomial: Conjugate Pair

For binary outcomes (Binomial likelihood), the Beta prior is conjugate:

Prior: p ~ Beta(α, β)
Data: x successes in n trials (Binomial)
Posterior: p | data ~ Beta(α + x, β + n - x)

The posterior is computed simply by adding observed successes to α and failures to β.

from scipy.stats import beta
Prior belief: Beta(2, 2) - mild skepticism

prior_alpha, prior_beta = 2, 2
Observe data

successes = 8
failures = 2
Update analytically

posterior_alpha = prior_alpha + successes
posterior_beta = prior_beta + failures
x = np.linspace(0, 1, 1000)
plt.figure(figsize=(10, 6))
Prior

plt.plot(x, beta.pdf(x, prior_alpha, prior_beta), 
         label=f'Prior: Beta({prior_alpha}, {prior_beta})', linewidth=2)
Posterior

plt.plot(x, beta.pdf(x, posterior_alpha, posterior_beta), 
         label=f'Posterior: Beta({posterior_alpha}, {posterior_beta})', linewidth=2)
plt.xlabel('Probability p')
plt.ylabel('Density')
plt.legend()
plt.title('Beta-Binomial Conjugacy: Analytical Update')
plt.grid(alpha=0.3)
plt.show()
Summary statistics

prior_mean = prior_alpha / (prior_alpha + prior_beta)
posterior_mean = posterior_alpha / (posterior_alpha + posterior_beta)
print(f"Prior mean: {prior_mean:.3f}")
print(f"Posterior mean: {posterior_mean:.3f}")
print(f"Posterior credible interval (95%): [{beta.ppf(0.025, posterior_alpha, posterior_beta):.3f}, {beta.ppf(0.975, posterior_alpha, posterior_beta):.3f}]")

MAP Estimation: Bridge to Frequentist

Maximum A Posteriori (MAP) estimation finds the parameter value that maximizes the posterior:

θ_MAP = argmax_θ p(θ|D) = argmax_θ p(D|θ) × p(θ)

When the prior is uniform (non-informative), MAP equals Maximum Likelihood Estimation (MLE).

MAP example with non-conjugate setup

np.random.seed(42)
Generate data: sample from Normal(5, 1)

data = np.random.normal(5, 1, 30)
Prior on mean: Normal(0, 10) - weakly informative

prior_mean, prior_std = 0, 10
Likelihood: Normal with known variance

likelihood_std = 1
Posterior parameters (closed form for Normal-Normal)

sample_mean = data.mean()
sample_std = data.std()
Posterior mean (weighted average of data mean and prior mean)

posterior_precision = (1/prior_std2) + (len(data)/likelihood_std2)
posterior_mean = ((sample_mean  len(data) / likelihood_std*2) + 
                  (prior_mean / prior_std2)) / posterior_precision
posterior_std = 1 / np.sqrt(posterior_precision)
print(f"Sample mean: {sample_mean:.3f}")
print(f"Prior mean: {prior_mean}")
print(f"Posterior mean (MAP for Normal prior): {posterior_mean:.3f}")
print(f"MLE (would ignore prior): {sample_mean:.3f}")
print(f"Posterior shrinkage brought estimate toward prior")

Bayesian vs. Frequentist: Key Differences

| Aspect | Bayesian | Frequentist | |--------|----------|-------------| | Parameters | Random variables | Fixed unknown constants | | Probability | Degree of belief | Long-run frequency | | Prior | Explicit, subjective | Absent | | Inference | Full posterior distribution | Point estimate + confidence interval | | Hypothesis test | P(H₀ \| data) | P(data \| H₀) |

Coin fairness: Bayesian vs Frequentist

from scipy.stats import binom_test
Observe 7 heads in 10 flips

data = 7
n = 10
Frequentist: p-value

p_value = binom_test(data, n, 0.5, alternative='two-sided')
print(f"Frequentist p-value: {p_value:.3f}")
Bayesian: probability coin is fair (p ∈ [0.4, 0.6])

posterior_alpha = 1 + data
posterior_beta = 1 + (n - data)
prob_fair = beta.cdf(0.6, posterior_alpha, posterior_beta) - beta.cdf(0.4, posterior_alpha, posterior_beta)
print(f"Bayesian P(p ∈ [0.4,0.6] | data): {prob_fair:.3f}")
print(f"\nFrequentist: p-value is P(data | H₀ is true)")
print(f"Bayesian: posterior directly gives P(H₀ | data)")

Why Bayesian Inference Matters for ML

1. Natural uncertainty quantification: Posteriors provide full distributions, not just point estimates 2. Principled regularization: Priors encode domain knowledge and prevent overfitting 3. Sequential learning: Posterior from one experiment becomes prior for the next 4. Decisions under uncertainty: Directly model what you care about—posterior on outcomes

Bayesian neural networks, Bayesian optimization, and probabilistic programming are increasingly central to modern machine learning.

Key Takeaways

  • Bayes' rule combines prior beliefs with data likelihood to produce posterior beliefs
  • Conjugate priors enable analytical updates for many common models
  • MAP estimation bridges Bayesian and frequentist approaches
  • Posterior distributions provide richer inference than point estimates alone

Quiz

1. In Bayes' rule p(θ|D) = p(D|θ)p(θ)/p(D), what does p(D|θ) represent?

  • A) Posterior probability
  • B) Prior probability
  • C) Likelihood ✓
  • D) Marginal likelihood

2. Beta-Binomial conjugacy means:

  • A) Beta priors produce Binomial posteriors
  • B) Both prior and posterior are Beta distributions ✓
  • C) Binomial and Beta are equivalent
  • D) Posterior has closed form for all priors

3. What is the relationship between MAP and MLE?

  • A) MAP and MLE are always equal
  • B) MAP equals MLE with uniform prior ✓
  • C) MLE is Bayesian, MAP is frequentist
  • D) They use different data

4. If you observe data inconsistent with your prior, the posterior will:

  • A) Ignore the data and stay like the prior
  • B) Move toward the data, weighted by prior strength ✓
  • C) Randomly combine prior and data
  • D) Become undefined

5. For Beta(2, 2) prior with 8 successes and 2 failures observed, the posterior is:

  • A) Beta(8, 2)
  • B) Beta(2, 2)
  • C) Beta(10, 4) ✓
  • D) Beta(6, 0)

6. Compared to frequentist confidence intervals, Bayesian credible intervals:

  • A) Have the same meaning
  • B) Directly state probability the parameter is in the interval ✓
  • C) Are always wider
  • D) Require larger samples

← Back ▶ Run in Colab