08 Naive Bayes — Exercises¶

Test your understanding of Bayes' theorem, posterior computation, Laplace smoothing, and the naive independence assumption.

Prerequisites. Read theory.md and work through first_principles.ipynb before attempting these.

In [1]:

import random

import matplotlib.pyplot as plt
import numpy as np

%load_ext autoreload
%autoreload 2

SEED = 42
random.seed(SEED)
rng = np.random.default_rng(SEED)

plt.rcParams["figure.dpi"] = 90

import random import matplotlib.pyplot as plt import numpy as np %load_ext autoreload %autoreload 2 SEED = 42 random.seed(SEED) rng = np.random.default_rng(SEED) plt.rcParams["figure.dpi"] = 90

Exercise 1 — Hand Calculation: Posterior Probability¶

A Gaussian Naive Bayes classifier has been trained on a 2-class, 2-feature problem. The learned parameters are:

Parameter	Class 0	Class 1
Prior $P(y=k)$	0.6	0.4
$\mu_{k1}$ (feature 1 mean)	2.0	5.0
$\sigma^2_{k1}$ (feature 1 variance)	1.0	2.0
$\mu_{k2}$ (feature 2 mean)	3.0	1.0
$\sigma^2_{k2}$ (feature 2 variance)	0.5	1.5

A test point has features $x = (3.0, 2.0)$.

Tasks (compute by hand, then verify numerically):

1. Compute $P(x_1 = 3.0 \mid y = k)$ for each class using the Gaussian density.
2. Compute $P(x_2 = 2.0 \mid y = k)$ for each class.
3. Compute the unnormalized posterior $P(y = k) \prod_j P(x_j \mid y = k)$ for each class.
4. Normalize to get $P(y = k \mid x)$ for each class.
5. Which class does the model predict?

Hint: Use $P(x_j \mid y = k) = \frac{1}{\sqrt{2\pi\sigma^2_{kj}}} \exp\left(-\frac{(x_j - \mu_{kj})^2}{2\sigma^2_{kj}}\right)$

Expected result: $P(y=0 \mid x) \approx 0.7567$, $P(y=1 \mid x) \approx 0.2433$, predict class 0.

In [2]:

# Verify your hand calculations
def gaussian_pdf(x, mu, var):
    """Univariate Gaussian density."""
    return (1.0 / np.sqrt(2 * np.pi * var)) * np.exp(-0.5 * (x - mu) ** 2 / var)


# Parameters
prior = np.array([0.6, 0.4])
mu = np.array([[2.0, 3.0],   # class 0: [mu_1, mu_2]
               [5.0, 1.0]])   # class 1: [mu_1, mu_2]
var = np.array([[1.0, 0.5],   # class 0: [var_1, var_2]
               [2.0, 1.5]])   # class 1: [var_1, var_2]
x_test = np.array([3.0, 2.0])

# Step 1-2: Compute likelihoods
likelihood = np.zeros(2)
for k in range(2):
    p_x1 = gaussian_pdf(x_test[0], mu[k, 0], var[k, 0])
    p_x2 = gaussian_pdf(x_test[1], mu[k, 1], var[k, 1])
    likelihood[k] = p_x1 * p_x2  # naive independence
    print(f'Class {k}: P(x1|y={k}) = {p_x1:.6f}, P(x2|y={k}) = {p_x2:.6f}, '
          f'product = {likelihood[k]:.6f}')

# Step 3: Unnormalized posterior
unnormalized = likelihood * prior
print(f'\nUnnormalized: class 0 = {unnormalized[0]:.6f}, class 1 = {unnormalized[1]:.6f}')

# Step 4: Normalize
posterior = unnormalized / unnormalized.sum()
print(f'Posterior: P(y=0|x) = {posterior[0]:.4f}, P(y=1|x) = {posterior[1]:.4f}')
print(f'Predicted class: {np.argmax(posterior)}')

# Check
assert np.isclose(posterior.sum(), 1.0, atol=1e-12)
assert np.isclose(posterior[0], 0.7567, atol=0.001), f'Expected ~0.7567, got {posterior[0]:.4f}'
assert np.argmax(posterior) == 0, 'Should predict class 0'
print('\nAll checks passed ✓')

# Verify your hand calculations def gaussian_pdf(x, mu, var): """Univariate Gaussian density.""" return (1.0 / np.sqrt(2 * np.pi * var)) * np.exp(-0.5 * (x - mu) ** 2 / var) # Parameters prior = np.array([0.6, 0.4]) mu = np.array([[2.0, 3.0], # class 0: [mu_1, mu_2] [5.0, 1.0]]) # class 1: [mu_1, mu_2] var = np.array([[1.0, 0.5], # class 0: [var_1, var_2] [2.0, 1.5]]) # class 1: [var_1, var_2] x_test = np.array([3.0, 2.0]) # Step 1-2: Compute likelihoods likelihood = np.zeros(2) for k in range(2): p_x1 = gaussian_pdf(x_test[0], mu[k, 0], var[k, 0]) p_x2 = gaussian_pdf(x_test[1], mu[k, 1], var[k, 1]) likelihood[k] = p_x1 * p_x2 # naive independence print(f'Class {k}: P(x1|y={k}) = {p_x1:.6f}, P(x2|y={k}) = {p_x2:.6f}, ' f'product = {likelihood[k]:.6f}') # Step 3: Unnormalized posterior unnormalized = likelihood * prior print(f'\nUnnormalized: class 0 = {unnormalized[0]:.6f}, class 1 = {unnormalized[1]:.6f}') # Step 4: Normalize posterior = unnormalized / unnormalized.sum() print(f'Posterior: P(y=0|x) = {posterior[0]:.4f}, P(y=1|x) = {posterior[1]:.4f}') print(f'Predicted class: {np.argmax(posterior)}') # Check assert np.isclose(posterior.sum(), 1.0, atol=1e-12) assert np.isclose(posterior[0], 0.7567, atol=0.001), f'Expected ~0.7567, got {posterior[0]:.4f}' assert np.argmax(posterior) == 0, 'Should predict class 0' print('\nAll checks passed ✓')

Class 0: P(x1|y=0) = 0.241971, P(x2|y=0) = 0.207554, product = 0.050222
Class 1: P(x1|y=1) = 0.103777, P(x2|y=1) = 0.233399, product = 0.024221

Unnormalized: class 0 = 0.030133, class 1 = 0.009689
Posterior: P(y=0|x) = 0.7567, P(y=1|x) = 0.2433
Predicted class: 0

All checks passed ✓

Exercise 2 — Coding: Laplace-Smoothed Multinomial NB¶

Implement a Multinomial Naive Bayes classifier with Laplace smoothing for text classification on a tiny bag-of-words dataset.

Tiny Dataset¶

Vocabulary: ['buy', 'cheap', 'free', 'money', 'hello', 'meeting', 'project']

Document	Words	Class
1	buy cheap free money	spam
2	free free money	spam
3	hello meeting project	ham
4	hello project project	ham
5	meeting meeting hello	ham

Requirements¶

1. Represent each document as a word-count vector (bag of words).
2. Implement MultinomialNBScratch with fit(X, y) and predict(X) methods.
3. Use Laplace smoothing with $\alpha = 1$.
4. Predict the class for a new document: "free money money".

Formulas¶

• $\hat{\theta}_{kj} = \frac{N_{kj} + \alpha}{N_k + \alpha \cdot d}$ where $N_{kj}$ = total count of word $j$ in class $k$, $N_k$ = total word count in class $k$, $d$ = vocabulary size.
• Log-posterior: $\log P(y=k) + \sum_j x_j \log \hat{\theta}_{kj}$

Deterministic check: "free money money" should be classified as spam.

In [3]:

class MultinomialNBScratch:
    """Multinomial Naive Bayes with Laplace smoothing."""

    def __init__(self, alpha: float = 1.0):
        self.alpha = alpha
        self.classes_ = None
        self.class_log_prior_ = None
        self.feature_log_prob_ = None  # shape (n_classes, n_features)

    def fit(self, X, y):
        """Estimate parameters from word-count matrix X and labels y.

        TODO: Implement the following steps:
        1. Identify unique classes and compute class priors.
        2. For each class, sum word counts across documents to get N_kj.
        3. Apply Laplace smoothing: theta_kj = (N_kj + alpha) / (N_k + alpha * d).
        4. Store log(theta_kj) in self.feature_log_prob_.
        """
        # TODO: implement
        pass

    def predict(self, X):
        """Predict class for each row in word-count matrix X.

        TODO: Compute log-posterior = log_prior + X @ feature_log_prob_.T
        and return the class with the highest log-posterior.
        """
        # TODO: implement
        pass

class MultinomialNBScratch: """Multinomial Naive Bayes with Laplace smoothing.""" def __init__(self, alpha: float = 1.0): self.alpha = alpha self.classes_ = None self.class_log_prior_ = None self.feature_log_prob_ = None # shape (n_classes, n_features) def fit(self, X, y): """Estimate parameters from word-count matrix X and labels y. TODO: Implement the following steps: 1. Identify unique classes and compute class priors. 2. For each class, sum word counts across documents to get N_kj. 3. Apply Laplace smoothing: theta_kj = (N_kj + alpha) / (N_k + alpha * d). 4. Store log(theta_kj) in self.feature_log_prob_. """ # TODO: implement pass def predict(self, X): """Predict class for each row in word-count matrix X. TODO: Compute log-posterior = log_prior + X @ feature_log_prob_.T and return the class with the highest log-posterior. """ # TODO: implement pass

Solution 2¶

In [4]:

class MultinomialNBScratch:
    """Multinomial Naive Bayes with Laplace smoothing (reference solution)."""

    def __init__(self, alpha: float = 1.0):
        self.alpha = alpha
        self.classes_ = None
        self.class_log_prior_ = None
        self.feature_log_prob_ = None  # shape (n_classes, n_features)

    def fit(self, X, y):
        """Estimate parameters from word-count matrix X and labels y."""
        X = np.asarray(X, dtype=float)
        y = np.asarray(y)
        d = X.shape[1]

        # 1. Unique classes and log priors
        self.classes_ = np.unique(y)
        n_classes = len(self.classes_)
        self.class_log_prior_ = np.zeros(n_classes)
        self.feature_log_prob_ = np.zeros((n_classes, d))

        for k, c in enumerate(self.classes_):
            Xk = X[y == c]
            self.class_log_prior_[k] = np.log(Xk.shape[0] / X.shape[0])
            # 2. Total count of each word in class k
            N_kj = Xk.sum(axis=0)
            N_k = N_kj.sum()
            # 3. Laplace smoothing: theta_kj = (N_kj + alpha) / (N_k + alpha * d)
            theta_k = (N_kj + self.alpha) / (N_k + self.alpha * d)
            # 4. Store log probabilities
            self.feature_log_prob_[k] = np.log(theta_k)
        return self

    def predict(self, X):
        """Predict class for each row in word-count matrix X."""
        X = np.asarray(X, dtype=float)
        # Log-posterior: log P(y=k) + sum_j x_j log theta_kj
        log_posterior = self.class_log_prior_ + X @ self.feature_log_prob_.T
        return self.classes_[np.argmax(log_posterior, axis=1)]

class MultinomialNBScratch: """Multinomial Naive Bayes with Laplace smoothing (reference solution).""" def __init__(self, alpha: float = 1.0): self.alpha = alpha self.classes_ = None self.class_log_prior_ = None self.feature_log_prob_ = None # shape (n_classes, n_features) def fit(self, X, y): """Estimate parameters from word-count matrix X and labels y.""" X = np.asarray(X, dtype=float) y = np.asarray(y) d = X.shape[1] # 1. Unique classes and log priors self.classes_ = np.unique(y) n_classes = len(self.classes_) self.class_log_prior_ = np.zeros(n_classes) self.feature_log_prob_ = np.zeros((n_classes, d)) for k, c in enumerate(self.classes_): Xk = X[y == c] self.class_log_prior_[k] = np.log(Xk.shape[0] / X.shape[0]) # 2. Total count of each word in class k N_kj = Xk.sum(axis=0) N_k = N_kj.sum() # 3. Laplace smoothing: theta_kj = (N_kj + alpha) / (N_k + alpha * d) theta_k = (N_kj + self.alpha) / (N_k + self.alpha * d) # 4. Store log probabilities self.feature_log_prob_[k] = np.log(theta_k) return self def predict(self, X): """Predict class for each row in word-count matrix X.""" X = np.asarray(X, dtype=float) # Log-posterior: log P(y=k) + sum_j x_j log theta_kj log_posterior = self.class_log_prior_ + X @ self.feature_log_prob_.T return self.classes_[np.argmax(log_posterior, axis=1)]

In [5]:

# Build the tiny dataset
# Vocabulary: ['buy', 'cheap', 'free', 'money', 'hello', 'meeting', 'project']
#              0       1        2       3        4        5          6

X_text = np.array([
    [1, 1, 1, 1, 0, 0, 0],  # doc 1: buy cheap free money -> spam
    [0, 0, 2, 1, 0, 0, 0],  # doc 2: free free money -> spam
    [0, 0, 0, 0, 1, 1, 1],  # doc 3: hello meeting project -> ham
    [0, 0, 0, 0, 1, 0, 2],  # doc 4: hello project project -> ham
    [0, 0, 0, 0, 1, 2, 0],  # doc 5: meeting meeting hello -> ham
], dtype=float)

y_text = np.array([1, 1, 0, 0, 0])  # 1 = spam, 0 = ham

# Train
mnb = MultinomialNBScratch(alpha=1.0)
mnb.fit(X_text, y_text)

# Predict on training data
train_preds = mnb.predict(X_text)
print(f'Training predictions: {train_preds}')
print(f'True labels:          {y_text}')
assert np.array_equal(train_preds, y_text), 'Should classify training data correctly'

# Predict new document: "free money money" -> [0, 0, 1, 2, 0, 0, 0]
X_new = np.array([[0, 0, 1, 2, 0, 0, 0]], dtype=float)
pred_new = mnb.predict(X_new)
print(f'\n"free money money" predicted as: {"spam" if pred_new[0] == 1 else "ham"}')
assert pred_new[0] == 1, '"free money money" should be classified as spam (class 1)'
print('All deterministic checks passed ✓')

# Build the tiny dataset # Vocabulary: ['buy', 'cheap', 'free', 'money', 'hello', 'meeting', 'project'] # 0 1 2 3 4 5 6 X_text = np.array([ [1, 1, 1, 1, 0, 0, 0], # doc 1: buy cheap free money -> spam [0, 0, 2, 1, 0, 0, 0], # doc 2: free free money -> spam [0, 0, 0, 0, 1, 1, 1], # doc 3: hello meeting project -> ham [0, 0, 0, 0, 1, 0, 2], # doc 4: hello project project -> ham [0, 0, 0, 0, 1, 2, 0], # doc 5: meeting meeting hello -> ham ], dtype=float) y_text = np.array([1, 1, 0, 0, 0]) # 1 = spam, 0 = ham # Train mnb = MultinomialNBScratch(alpha=1.0) mnb.fit(X_text, y_text) # Predict on training data train_preds = mnb.predict(X_text) print(f'Training predictions: {train_preds}') print(f'True labels: {y_text}') assert np.array_equal(train_preds, y_text), 'Should classify training data correctly' # Predict new document: "free money money" -> [0, 0, 1, 2, 0, 0, 0] X_new = np.array([[0, 0, 1, 2, 0, 0, 0]], dtype=float) pred_new = mnb.predict(X_new) print(f'\n"free money money" predicted as: {"spam" if pred_new[0] == 1 else "ham"}') assert pred_new[0] == 1, '"free money money" should be classified as spam (class 1)' print('All deterministic checks passed ✓')

Training predictions: [1 1 0 0 0]
True labels:          [1 1 0 0 0]

"free money money" predicted as: spam
All deterministic checks passed ✓

Bonus: Verify Against sklearn¶

In [6]:

# After implementing, uncomment to compare with sklearn:
#
# from sklearn.naive_bayes import MultinomialNB as SkMNB
# sk_mnb = SkMNB(alpha=1.0)
# sk_mnb.fit(X_text, y_text)
# sk_pred_new = sk_mnb.predict(X_new)
# print(f'sklearn prediction for "free money money": {sk_pred_new[0]}')
# assert np.array_equal(pred_new, sk_pred_new), 'Predictions should match sklearn'

# After implementing, uncomment to compare with sklearn: # # from sklearn.naive_bayes import MultinomialNB as SkMNB # sk_mnb = SkMNB(alpha=1.0) # sk_mnb.fit(X_text, y_text) # sk_pred_new = sk_mnb.predict(X_new) # print(f'sklearn prediction for "free money money": {sk_pred_new[0]}') # assert np.array_equal(pred_new, sk_pred_new), 'Predictions should match sklearn'

Exercise 3 — Conceptual: Why Does Naive Bayes Work Despite the Wrong Assumption?¶

The naive conditional independence assumption $P(x \mid y) = \prod_j P(x_j \mid y)$ is almost never true in real data. Word co-occurrences are clearly dependent ("machine" and "learning" co-occur), pixel values are spatially correlated, etc.

Questions:

1. Classification vs. estimation: Naive Bayes is known to be a poor probability estimator but often a good classifier. Explain why. What property of the decision rule $\arg\max_k P(y=k \mid x)$ makes it robust to biased probability estimates?

2. Ng & Jordan (2001) result: The generative–discriminative pair theory shows that Naive Bayes converges to its (possibly suboptimal) asymptotic accuracy faster than logistic regression. In what practical scenarios (dataset size, feature dimensionality) would you choose NB over logistic regression?

3. Double-counting: If you have two features $x_1$ and $x_2$ that are perfectly correlated ($x_2 = x_1$), how does Naive Bayes treat them? What happens to the posterior probability estimates? Does the predicted class change?

4. Laplace smoothing as a prior: Show that Laplace smoothing with $\alpha = 1$ in Multinomial NB is equivalent to placing a $\text{Dir}(1, 1, \dots, 1)$ (uniform Dirichlet) prior on the word probabilities and computing the MAP estimate.

In [7]:

# Numerical exploration for Q3: duplicate features
# Use the GaussianNB from first_principles.ipynb or from src/
from ml_first_principles.probabilistic_models import GaussianNB

# Simple 1D dataset
X_orig = np.array([[1.0], [2.0], [3.0], [7.0], [8.0], [9.0]])
y_orig = np.array([0, 0, 0, 1, 1, 1])

# Duplicate the feature
X_dup = np.hstack([X_orig, X_orig])  # x2 = x1 exactly

# Train both
gnb_1d = GaussianNB(var_smoothing=1e-9)
gnb_1d.fit(X_orig, y_orig)

gnb_2d = GaussianNB(var_smoothing=1e-9)
gnb_2d.fit(X_dup, y_orig)

# Compare predictions at x=5 (ambiguous point)
x_ambiguous_1d = np.array([[5.0]])
x_ambiguous_2d = np.array([[5.0, 5.0]])

# TODO: Compare the joint log-likelihoods and discuss
# What happens to the confidence? Does the predicted class change?
# jll_1d = gnb_1d._joint_log_likelihood(x_ambiguous_1d)
# jll_2d = gnb_2d._joint_log_likelihood(x_ambiguous_2d)
# print(f'Log-likelihoods (1 feature): {jll_1d}')
# print(f'Log-likelihoods (2 features): {jll_2d}')
# print(f'Gap doubles but predicted class is the same.')

# Numerical exploration for Q3: duplicate features # Use the GaussianNB from first_principles.ipynb or from src/ from ml_first_principles.probabilistic_models import GaussianNB # Simple 1D dataset X_orig = np.array([[1.0], [2.0], [3.0], [7.0], [8.0], [9.0]]) y_orig = np.array([0, 0, 0, 1, 1, 1]) # Duplicate the feature X_dup = np.hstack([X_orig, X_orig]) # x2 = x1 exactly # Train both gnb_1d = GaussianNB(var_smoothing=1e-9) gnb_1d.fit(X_orig, y_orig) gnb_2d = GaussianNB(var_smoothing=1e-9) gnb_2d.fit(X_dup, y_orig) # Compare predictions at x=5 (ambiguous point) x_ambiguous_1d = np.array([[5.0]]) x_ambiguous_2d = np.array([[5.0, 5.0]]) # TODO: Compare the joint log-likelihoods and discuss # What happens to the confidence? Does the predicted class change? # jll_1d = gnb_1d._joint_log_likelihood(x_ambiguous_1d) # jll_2d = gnb_2d._joint_log_likelihood(x_ambiguous_2d) # print(f'Log-likelihoods (1 feature): {jll_1d}') # print(f'Log-likelihoods (2 features): {jll_2d}') # print(f'Gap doubles but predicted class is the same.')