04 Logistic Regression — Exercises¶

Test your understanding of the sigmoid, cross-entropy, gradient derivation, decision boundaries, and multi-class extension.

Prerequisites. Read theory.md and work through first_principles.ipynb before attempting these.

In [1]:

import random

import matplotlib.pyplot as plt
import numpy as np

%load_ext autoreload
%autoreload 2

SEED = 42
random.seed(SEED)
rng = np.random.default_rng(SEED)

plt.rcParams["figure.dpi"] = 90

import random import matplotlib.pyplot as plt import numpy as np %load_ext autoreload %autoreload 2 SEED = 42 random.seed(SEED) rng = np.random.default_rng(SEED) plt.rcParams["figure.dpi"] = 90

Exercise 1 — Hand Derivation: Sigmoid Identities¶

Let $\sigma(z) = 1 / (1 + e^{-z})$.

Tasks (derive by hand, then verify numerically):

1. Prove the reflection identity: $\sigma(-z) = 1 - \sigma(z)$.
2. Prove the derivative identity: $\sigma'(z) = \sigma(z)(1 - \sigma(z))$.
3. At what value of $z$ is $\sigma'(z)$ maximized? What is that maximum value?

Expected numerical checks:

$z$	$\sigma(z)$	$\sigma(-z)$	$\sigma(z) + \sigma(-z)$	$\sigma'(z)$
0.0	0.5000	0.5000	1.0000	0.2500
2.0	0.8808	0.1192	1.0000	0.1050
-3.0	0.0474	0.9526	1.0000	0.0452

In [2]:

def sigmoid(z):
    return 1.0 / (1.0 + np.exp(-z))


# Verify identities numerically
for z in [0.0, 2.0, -3.0, 10.0, -10.0]:
    s = sigmoid(z)
    s_neg = sigmoid(-z)
    s_prime = s * (1 - s)
    # (I1) reflection
    assert np.isclose(s + s_neg, 1.0, atol=1e-12), f'Reflection failed at z={z}'
    # (I2) derivative via finite difference
    eps = 1e-7
    fd = (sigmoid(z + eps) - sigmoid(z - eps)) / (2 * eps)
    assert np.isclose(s_prime, fd, atol=1e-6), f'Derivative failed at z={z}'

# Maximum of sigma'(z) occurs at z=0
assert np.isclose(sigmoid(0) * (1 - sigmoid(0)), 0.25, atol=1e-12)
print('All sigmoid identity checks passed.')

def sigmoid(z): return 1.0 / (1.0 + np.exp(-z)) # Verify identities numerically for z in [0.0, 2.0, -3.0, 10.0, -10.0]: s = sigmoid(z) s_neg = sigmoid(-z) s_prime = s * (1 - s) # (I1) reflection assert np.isclose(s + s_neg, 1.0, atol=1e-12), f'Reflection failed at z={z}' # (I2) derivative via finite difference eps = 1e-7 fd = (sigmoid(z + eps) - sigmoid(z - eps)) / (2 * eps) assert np.isclose(s_prime, fd, atol=1e-6), f'Derivative failed at z={z}' # Maximum of sigma'(z) occurs at z=0 assert np.isclose(sigmoid(0) * (1 - sigmoid(0)), 0.25, atol=1e-12) print('All sigmoid identity checks passed.')

All sigmoid identity checks passed.

Exercise 2 — Hand Calculation: Cross-Entropy Loss¶

A model outputs the following probabilities for 4 examples:

$i$	$y_i$	$\hat{p}_i$
1	1	0.9
2	0	0.2
3	1	0.4
4	0	0.8

Tasks:

1. Compute the per-example loss $\ell_i = -[y_i \log \hat{p}_i + (1-y_i) \log(1-\hat{p}_i)]$ for each example.
2. Which example has the highest loss? Why does this make sense?
3. Compute the average cross-entropy loss $L = \frac{1}{4}\sum_i \ell_i$.

Expected result: $L \approx 0.6319$

In [3]:

# Verify your hand calculations
y = np.array([1, 0, 1, 0])
p_hat = np.array([0.9, 0.2, 0.4, 0.8])

# TODO: compute per-example loss and average
# per_example = ...
# L = ...
# print(f'Per-example losses: {per_example}')
# print(f'Average loss: {L:.4f}')
# assert np.isclose(L, 0.6319, atol=0.001)

# Verify your hand calculations y = np.array([1, 0, 1, 0]) p_hat = np.array([0.9, 0.2, 0.4, 0.8]) # TODO: compute per-example loss and average # per_example = ... # L = ... # print(f'Per-example losses: {per_example}') # print(f'Average loss: {L:.4f}') # assert np.isclose(L, 0.6319, atol=0.001)

Exercise 3 — Coding: Gradient Descent Step¶

Implement a single gradient descent update for logistic regression.

Recall the gradient: $\nabla L(\theta) = \frac{1}{n} X^T (p - y)$ where $p_i = \sigma(x_i^T \theta)$.

Tasks:

1. Implement logistic_gradient(X, y, theta) that returns the gradient.
2. Implement logistic_loss(X, y, theta) that returns the average cross-entropy.
3. Verify on a small dataset: 3 points in 2D (with bias column).

Test dataset:

X = [[1, 0],    y = [0,
     [1, 1],         0,
     [1, 3]]         1]
theta = [0, 0]

Expected results:

• Initial loss: $\log(2) \approx 0.6931$ (since $\sigma(0) = 0.5$ for all)
• Gradient: $[1/6, \; -1/3] \approx [0.1667, \; -0.3333]$ (hint: $p - y = [0.5, 0.5, -0.5]$, then $(1/3) X^T (p - y)$)

In [4]:

def logistic_loss(X, y, theta):
    """Average binary cross-entropy loss."""
    # TODO: implement
    pass


def logistic_gradient(X, y, theta):
    """Gradient of average cross-entropy: (1/n) X^T (sigma(X @ theta) - y)."""
    # TODO: implement
    pass

def logistic_loss(X, y, theta): """Average binary cross-entropy loss.""" # TODO: implement pass def logistic_gradient(X, y, theta): """Gradient of average cross-entropy: (1/n) X^T (sigma(X @ theta) - y).""" # TODO: implement pass

Solution 3¶

In [5]:

def logistic_loss(X, y, theta):
    """Average binary cross-entropy loss."""
    p = sigmoid(X @ theta)
    return float(-np.mean(y * np.log(p) + (1 - y) * np.log(1 - p)))


def logistic_gradient(X, y, theta):
    """Gradient of average cross-entropy: (1/n) X^T (sigma(X @ theta) - y)."""
    p = sigmoid(X @ theta)
    return X.T @ (p - y) / len(y)


# Deterministic check
X_ex3 = np.array([[1, 0], [1, 1], [1, 3]], dtype=float)
y_ex3 = np.array([0, 0, 1], dtype=float)
theta_ex3 = np.array([0.0, 0.0])

loss_0 = logistic_loss(X_ex3, y_ex3, theta_ex3)
grad_0 = logistic_gradient(X_ex3, y_ex3, theta_ex3)

assert np.isclose(loss_0, np.log(2), atol=1e-10), f'Loss should be log(2), got {loss_0}'
assert np.allclose(grad_0, [1/6, -1/3], atol=1e-10), f'Gradient wrong: {grad_0}'
print(f'Loss at theta=0: {loss_0:.4f} (expected {np.log(2):.4f})')
print(f'Gradient at theta=0: {grad_0} (expected [0.1667, -0.3333])')
print('All gradient checks passed.')

def logistic_loss(X, y, theta): """Average binary cross-entropy loss.""" p = sigmoid(X @ theta) return float(-np.mean(y * np.log(p) + (1 - y) * np.log(1 - p))) def logistic_gradient(X, y, theta): """Gradient of average cross-entropy: (1/n) X^T (sigma(X @ theta) - y).""" p = sigmoid(X @ theta) return X.T @ (p - y) / len(y) # Deterministic check X_ex3 = np.array([[1, 0], [1, 1], [1, 3]], dtype=float) y_ex3 = np.array([0, 0, 1], dtype=float) theta_ex3 = np.array([0.0, 0.0]) loss_0 = logistic_loss(X_ex3, y_ex3, theta_ex3) grad_0 = logistic_gradient(X_ex3, y_ex3, theta_ex3) assert np.isclose(loss_0, np.log(2), atol=1e-10), f'Loss should be log(2), got {loss_0}' assert np.allclose(grad_0, [1/6, -1/3], atol=1e-10), f'Gradient wrong: {grad_0}' print(f'Loss at theta=0: {loss_0:.4f} (expected {np.log(2):.4f})') print(f'Gradient at theta=0: {grad_0} (expected [0.1667, -0.3333])') print('All gradient checks passed.')

Loss at theta=0: 0.6931 (expected 0.6931)
Gradient at theta=0: [ 0.16666667 -0.33333333] (expected [0.1667, -0.3333])
All gradient checks passed.

Exercise 4 — Coding: Decision Boundary Visualization¶

Generate a 2D binary classification dataset (two Gaussian blobs) and train a logistic regression classifier on it.

Tasks:

1. Generate 200 points: class 0 centered at $(-1, -1)$, class 1 centered at $(1, 1)$, each with standard deviation 1.0.
2. Train your logistic regression from first_principles.ipynb (or from ml_first_principles.linear_models).
3. Plot the data points (colored by class) and overlay the decision boundary (the line where $\theta^T x = 0$).
4. Also plot filled contours showing the predicted probability $P(y=1 \mid x)$.

Hint: The decision boundary is a line satisfying $\theta_0 + \theta_1 x_1 + \theta_2 x_2 = 0$, i.e. $x_2 = -(\theta_0 + \theta_1 x_1) / \theta_2$.

In [6]:

# TODO: generate data, train classifier, plot boundary + probability contours
# n_per_class = 100
# X0 = rng.normal(loc=[-1, -1], scale=1.0, size=(n_per_class, 2))
# X1 = rng.normal(loc=[1, 1], scale=1.0, size=(n_per_class, 2))
# ...

# TODO: generate data, train classifier, plot boundary + probability contours # n_per_class = 100 # X0 = rng.normal(loc=[-1, -1], scale=1.0, size=(n_per_class, 2)) # X1 = rng.normal(loc=[1, 1], scale=1.0, size=(n_per_class, 2)) # ...

Exercise 5 — Conceptual: Logistic vs Linear Regression¶

Questions:

1. Both linear and logistic regression have gradient of the form $(1/n) X^T r$. What is the residual $r$ in each case? Why does the logistic residual stay bounded in $(-1, 1)$?

2. Linear regression has a closed-form solution. Why can't the same approach work for logistic regression? What specific mathematical obstruction prevents isolating $\theta$?

3. A colleague uses OLS (ordinary least squares) for a binary classification problem and gets "reasonable" accuracy. Under what conditions might OLS give similar predictions to logistic regression? When would it fail badly?

Exercise 6 — Failure Analysis: Perfect Separation¶

When data is perfectly linearly separable, the unregularised MLE does not exist — $\|\theta\| \to \infty$.

Tasks:

1. Create a perfectly separable 1D dataset: class 0 at $x \in \{1, 2, 3\}$, class 1 at $x \in \{5, 6, 7\}$.
2. Run gradient descent for 5000 iterations with $\eta = 0.1$. Plot $\|\theta\|_2$ vs iteration. Does it converge or keep growing?
3. Add L2 regularization ($\lambda = 0.1$) and repeat. Does $\|\theta\|$ stabilize?

Deterministic check: Without regularization, $\|\theta\|$ should exceed 10.0 after 5000 iterations. With regularization, it should stay below 5.0.

In [7]:

# TODO: create separable data, run GD with and without regularization,
# plot ||theta|| over iterations
# X_sep = np.column_stack([np.ones(6), [1, 2, 3, 5, 6, 7]])
# y_sep = np.array([0, 0, 0, 1, 1, 1], dtype=float)
# ...

# TODO: create separable data, run GD with and without regularization, # plot ||theta|| over iterations # X_sep = np.column_stack([np.ones(6), [1, 2, 3, 5, 6, 7]]) # y_sep = np.array([0, 0, 0, 1, 1, 1], dtype=float) # ...

Exercise 7 — Conceptual: Log-Odds and Coefficient Interpretation¶

A logistic regression model for loan default has these coefficients:

Feature	$\theta_j$
intercept	-2.0
income ($10\text{k}$)	-0.5
debt_ratio	1.2

Questions:

1. For a person with income = $50\text{k}$ and debt_ratio = 0.3, compute the log-odds, odds, and probability of default.

2. If income increases by $10\text{k}$ (1 unit), by what factor do the odds of default change? Does more income increase or decrease default risk?

3. If debt_ratio increases by 0.1, by what factor do the odds change?

Expected results for Q1:

• Log-odds: $-2.0 + (-0.5)(5) + 1.2(0.3) = -4.14$
• Odds: $e^{-4.14} \approx 0.0160$
• Probability: $\sigma(-4.14) \approx 0.0158$