13 Neural Networks — Exercises¶

Test your understanding of forward propagation, backpropagation, gradient computation, activation functions, and MLP capacity.

Prerequisites. Read theory.md and work through first_principles.ipynb before attempting these.

In [1]:

Copied!





import random

import matplotlib.pyplot as plt
import numpy as np

%load_ext autoreload
%autoreload 2

SEED = 42
random.seed(SEED)
rng = np.random.default_rng(SEED)
import random

import matplotlib.pyplot as plt
import numpy as np

%load_ext autoreload
%autoreload 2

SEED = 42
random.seed(SEED)
rng = np.random.default_rng(SEED)

Exercise 1 — Hand Calculation: Forward Pass and Backprop¶

Consider a tiny network: 2 inputs → 1 hidden unit (sigmoid) → 1 output (sigmoid).

Architecture:

$$x_1, x_2 \xrightarrow{w_1, w_2, b_1} z_1 \xrightarrow{\sigma} h \xrightarrow{w_3, b_2} z_2 \xrightarrow{\sigma} \hat{y}$$

Given weights and input:

Parameter	Value
$w_1$	0.5
$w_2$	-0.3
$b_1$	0.1
$w_3$	0.8
$b_2$	-0.2
$x_1$	1.0
$x_2$	2.0
$y$ (true label)	1

Recall: $\sigma(z) = 1/(1+e^{-z})$ and $\sigma'(z) = \sigma(z)(1-\sigma(z))$.

Tasks (derive by hand, then verify numerically):

Forward pass: Compute $z_1$, $h = \sigma(z_1)$, $z_2$, $\hat{y} = \sigma(z_2)$.
Loss: Compute the binary cross-entropy $L = -[y \log \hat{y} + (1-y)\log(1-\hat{y})]$.
Backprop: Compute $\frac{\partial L}{\partial w_3}$, $\frac{\partial L}{\partial b_2}$, $\frac{\partial L}{\partial w_1}$, $\frac{\partial L}{\partial w_2}$, $\frac{\partial L}{\partial b_1}$.

Backprop chain rule:

$$\frac{\partial L}{\partial z_2} = \hat{y} - y \qquad \text{(softmax/sigmoid + CE shortcut)}$$

$$\frac{\partial L}{\partial w_3} = \frac{\partial L}{\partial z_2} \cdot h, \quad \frac{\partial L}{\partial b_2} = \frac{\partial L}{\partial z_2}$$

$$\frac{\partial L}{\partial h} = \frac{\partial L}{\partial z_2} \cdot w_3$$

$$\frac{\partial L}{\partial z_1} = \frac{\partial L}{\partial h} \cdot \sigma'(z_1) = \frac{\partial L}{\partial h} \cdot h(1-h)$$

$$\frac{\partial L}{\partial w_1} = \frac{\partial L}{\partial z_1} \cdot x_1, \quad \frac{\partial L}{\partial w_2} = \frac{\partial L}{\partial z_1} \cdot x_2, \quad \frac{\partial L}{\partial b_1} = \frac{\partial L}{\partial z_1}$$

Expected results (verify to 4 d.p.):

Quantity	Value
$z_1$	0.0000
$h$	0.5000
$z_2$	0.2000
$\hat{y}$	0.5498
$L$	0.5981
$\partial L/\partial w_3$	−0.2251
$\partial L/\partial b_2$	−0.4502
$\partial L/\partial w_1$	−0.0900
$\partial L/\partial w_2$	−0.1801
$\partial L/\partial b_1$	−0.0900

In [2]:

Copied!





def sigmoid(z):
    return 1.0 / (1.0 + np.exp(-z))


# Given values
x1, x2 = 1.0, 2.0
w1, w2, b1 = 0.5, -0.3, 0.1
w3, b2 = 0.8, -0.2
y_true = 1.0

# TODO: Forward pass
# z1 = ...
# h = ...
# z2 = ...
# y_hat = ...

# TODO: Loss
# L = ...

# TODO: Backprop
# dL_dz2 = ...
# dL_dw3 = ...
# dL_db2 = ...
# dL_dh = ...
# dL_dz1 = ...
# dL_dw1 = ...
# dL_dw2 = ...
# dL_db1 = ...

# Uncomment to verify:
# assert np.isclose(z1, 0.0, atol=1e-10)
# assert np.isclose(h, 0.5, atol=1e-4)
# assert np.isclose(z2, 0.2, atol=1e-10)
# assert np.isclose(y_hat, 0.5498, atol=1e-4)
# assert np.isclose(L, 0.5981, atol=1e-4)
# assert np.isclose(dL_dw3, -0.2251, atol=1e-4)
# assert np.isclose(dL_db2, -0.4502, atol=1e-4)
# assert np.isclose(dL_dw1, -0.0900, atol=1e-4)
# assert np.isclose(dL_dw2, -0.1801, atol=1e-4)
# assert np.isclose(dL_db1, -0.0900, atol=1e-4)
# print('All hand-calculation checks passed. ✓')
def sigmoid(z):
    return 1.0 / (1.0 + np.exp(-z))


# Given values
x1, x2 = 1.0, 2.0
w1, w2, b1 = 0.5, -0.3, 0.1
w3, b2 = 0.8, -0.2
y_true = 1.0

# TODO: Forward pass
# z1 = ...
# h = ...
# z2 = ...
# y_hat = ...

# TODO: Loss
# L = ...

# TODO: Backprop
# dL_dz2 = ...
# dL_dw3 = ...
# dL_db2 = ...
# dL_dh = ...
# dL_dz1 = ...
# dL_dw1 = ...
# dL_dw2 = ...
# dL_db1 = ...

# Uncomment to verify:
# assert np.isclose(z1, 0.0, atol=1e-10)
# assert np.isclose(h, 0.5, atol=1e-4)
# assert np.isclose(z2, 0.2, atol=1e-10)
# assert np.isclose(y_hat, 0.5498, atol=1e-4)
# assert np.isclose(L, 0.5981, atol=1e-4)
# assert np.isclose(dL_dw3, -0.2251, atol=1e-4)
# assert np.isclose(dL_db2, -0.4502, atol=1e-4)
# assert np.isclose(dL_dw1, -0.0900, atol=1e-4)
# assert np.isclose(dL_dw2, -0.1801, atol=1e-4)
# assert np.isclose(dL_db1, -0.0900, atol=1e-4)
# print('All hand-calculation checks passed. ✓')

Exercise 2 — Coding: Dense Layer Forward + Backward with Gradient Checking¶

Implement a single dense (fully-connected) layer that computes:

Forward: $Z = XW + b$ where $X \in \mathbb{R}^{n \times d_{\text{in}}}$, $W \in \mathbb{R}^{d_{\text{in}} \times d_{\text{out}}}$, $b \in \mathbb{R}^{d_{\text{out}}}$.

Backward: Given upstream gradient $G_Z = \frac{\partial L}{\partial Z} \in \mathbb{R}^{n \times d_{\text{out}}}$, compute:

$$\frac{\partial L}{\partial W} = X^T G_Z, \quad \frac{\partial L}{\partial b} = \sum_{i=1}^{n} (G_Z)_i = \mathbf{1}^T G_Z, \quad \frac{\partial L}{\partial X} = G_Z W^T$$

Tasks:

Implement the DenseLayer class with forward(X) and backward(G_Z) methods.
Verify all three gradients ($\partial L/\partial W$, $\partial L/\partial b$, $\partial L/\partial X$) using finite differences on a small random example.

Deterministic check: All gradient relative errors must be < $10^{-5}$.

In [3]:

Copied!





class DenseLayer:
    """Single dense layer: Z = X @ W + b."""

    def __init__(self, d_in, d_out, rng):
        scale = np.sqrt(2.0 / (d_in + d_out))
        self.W = rng.normal(0, scale, size=(d_in, d_out))
        self.b = np.zeros(d_out)
        # Cache for backward
        self.X_cache = None

    def forward(self, X):
        """Compute Z = X @ W + b. Cache X for backward."""
        # TODO: implement
        # self.X_cache = ...
        # return ...
        pass

    def backward(self, G_Z):
        """Given dL/dZ, compute dL/dW, dL/db, dL/dX.

        Returns:
            G_X: gradient w.r.t. input X (to pass upstream)
        """
        # TODO: implement
        # self.dW = ...
        # self.db = ...
        # G_X = ...
        # return G_X
        pass
class DenseLayer:
    """Single dense layer: Z = X @ W + b."""

    def __init__(self, d_in, d_out, rng):
        scale = np.sqrt(2.0 / (d_in + d_out))
        self.W = rng.normal(0, scale, size=(d_in, d_out))
        self.b = np.zeros(d_out)
        # Cache for backward
        self.X_cache = None

    def forward(self, X):
        """Compute Z = X @ W + b. Cache X for backward."""
        # TODO: implement
        # self.X_cache = ...
        # return ...
        pass

    def backward(self, G_Z):
        """Given dL/dZ, compute dL/dW, dL/db, dL/dX.

        Returns:
            G_X: gradient w.r.t. input X (to pass upstream)
        """
        # TODO: implement
        # self.dW = ...
        # self.db = ...
        # G_X = ...
        # return G_X
        pass

In [4]:

Copied!





# Gradient check via finite differences
rng_ex2 = np.random.default_rng(99)
n_ex2, d_in_ex2, d_out_ex2 = 5, 3, 2
layer = DenseLayer(d_in_ex2, d_out_ex2, rng_ex2)
X_ex2 = rng_ex2.normal(size=(n_ex2, d_in_ex2))

# Use sum of outputs as a simple scalar loss: L = sum(Z)
# so dL/dZ = ones_like(Z)
Z_ex2 = layer.forward(X_ex2)
G_Z_ex2 = np.ones_like(Z_ex2)  # dL/dZ for L = sum(Z)
G_X_ex2 = layer.backward(G_Z_ex2)

eps = 1e-5


def numerical_grad(param, compute_loss):
    """Finite-difference gradient for a parameter array."""
    grad = np.zeros_like(param)
    it = np.nditer(param, flags=['multi_index'])
    while not it.finished:
        idx = it.multi_index
        old = param[idx]
        param[idx] = old + eps
        loss_plus = compute_loss()
        param[idx] = old - eps
        loss_minus = compute_loss()
        grad[idx] = (loss_plus - loss_minus) / (2 * eps)
        param[idx] = old
        it.iternext()
    return grad


# TODO: compute numerical gradients for W, b, and X
# dW_num = numerical_grad(layer.W, lambda: layer.forward(X_ex2).sum())
# db_num = numerical_grad(layer.b, lambda: layer.forward(X_ex2).sum())
# dX_num = numerical_grad(X_ex2, lambda: layer.forward(X_ex2).sum())

# TODO: compare with analytic gradients
# assert np.allclose(layer.dW, dW_num, atol=1e-5), 'dW mismatch'
# assert np.allclose(layer.db, db_num, atol=1e-5), 'db mismatch'
# assert np.allclose(G_X_ex2, dX_num, atol=1e-5), 'dX mismatch'
# print('All Dense layer gradient checks passed. ✓')
# Gradient check via finite differences
rng_ex2 = np.random.default_rng(99)
n_ex2, d_in_ex2, d_out_ex2 = 5, 3, 2
layer = DenseLayer(d_in_ex2, d_out_ex2, rng_ex2)
X_ex2 = rng_ex2.normal(size=(n_ex2, d_in_ex2))

# Use sum of outputs as a simple scalar loss: L = sum(Z)
# so dL/dZ = ones_like(Z)
Z_ex2 = layer.forward(X_ex2)
G_Z_ex2 = np.ones_like(Z_ex2)  # dL/dZ for L = sum(Z)
G_X_ex2 = layer.backward(G_Z_ex2)

eps = 1e-5


def numerical_grad(param, compute_loss):
    """Finite-difference gradient for a parameter array."""
    grad = np.zeros_like(param)
    it = np.nditer(param, flags=['multi_index'])
    while not it.finished:
        idx = it.multi_index
        old = param[idx]
        param[idx] = old + eps
        loss_plus = compute_loss()
        param[idx] = old - eps
        loss_minus = compute_loss()
        grad[idx] = (loss_plus - loss_minus) / (2 * eps)
        param[idx] = old
        it.iternext()
    return grad


# TODO: compute numerical gradients for W, b, and X
# dW_num = numerical_grad(layer.W, lambda: layer.forward(X_ex2).sum())
# db_num = numerical_grad(layer.b, lambda: layer.forward(X_ex2).sum())
# dX_num = numerical_grad(X_ex2, lambda: layer.forward(X_ex2).sum())

# TODO: compare with analytic gradients
# assert np.allclose(layer.dW, dW_num, atol=1e-5), 'dW mismatch'
# assert np.allclose(layer.db, db_num, atol=1e-5), 'db mismatch'
# assert np.allclose(G_X_ex2, dX_num, atol=1e-5), 'dX mismatch'
# print('All Dense layer gradient checks passed. ✓')

Exercise 3 — Conceptual: Why Can't a Single Layer Solve XOR?¶

The XOR function maps:

$x_1$	$x_2$	$y$
0	0	0
0	1	1
1	0	1
1	1	0

Questions:

A single-layer network (no hidden layer) computes $\hat{y} = \sigma(w_1 x_1 + w_2 x_2 + b)$. The decision boundary is the line $w_1 x_1 + w_2 x_2 + b = 0$. Draw the four XOR points on a 2D plane and argue geometrically why no single line can separate the two classes.
What is the minimum architecture (number of layers, units per layer) needed to solve XOR exactly? Justify your answer.
Bonus: Construct explicit weights for a 2-input, 2-hidden (sigmoid), 1-output (sigmoid) network that computes XOR. Verify numerically below.

Hint for the bonus: Think of XOR as $(x_1 \text{ AND NOT } x_2) \text{ OR } (x_2 \text{ AND NOT } x_1)$. Each hidden unit can implement one AND gate.

In [5]:

Copied!





# Geometric visualization: plot XOR points
X_xor_exact = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
y_xor_exact = np.array([0, 1, 1, 0])

fig, ax = plt.subplots(figsize=(5, 5))
for c, marker, label in [(0, 'o', 'class 0'), (1, 's', 'class 1')]:
    mask = y_xor_exact == c
    ax.scatter(X_xor_exact[mask, 0], X_xor_exact[mask, 1],
               s=120, marker=marker, label=label, edgecolor='k', linewidth=1.5, zorder=5)
ax.set_xlabel('$x_1$')
ax.set_ylabel('$x_2$')
ax.set_title('XOR — no single line separates the classes')
ax.set_xlim(-0.5, 1.5)
ax.set_ylim(-0.5, 1.5)
ax.legend()
ax.grid(True, alpha=0.3)
plt.show()

# TODO (bonus): construct explicit weights for a 2-2-1 network that computes XOR
# W1_xor = np.array(...)  # shape (2, 2)
# b1_xor = np.array(...)  # shape (2,)
# W2_xor = np.array(...)  # shape (2, 1)
# b2_xor = np.array(...)  # shape (1,)
#
# Z1 = X_xor_exact @ W1_xor + b1_xor
# H = sigmoid(Z1)
# Z2 = H @ W2_xor + b2_xor
# out = sigmoid(Z2)
# preds = (out.ravel() > 0.5).astype(int)
# assert np.array_equal(preds, y_xor_exact), f'XOR failed: {preds}'
# print(f'Hand-crafted XOR network outputs: {out.ravel()}')
# print(f'Predictions: {preds} == {y_xor_exact} ✓')
# Geometric visualization: plot XOR points
X_xor_exact = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
y_xor_exact = np.array([0, 1, 1, 0])

fig, ax = plt.subplots(figsize=(5, 5))
for c, marker, label in [(0, 'o', 'class 0'), (1, 's', 'class 1')]:
    mask = y_xor_exact == c
    ax.scatter(X_xor_exact[mask, 0], X_xor_exact[mask, 1],
               s=120, marker=marker, label=label, edgecolor='k', linewidth=1.5, zorder=5)
ax.set_xlabel('$x_1$')
ax.set_ylabel('$x_2$')
ax.set_title('XOR — no single line separates the classes')
ax.set_xlim(-0.5, 1.5)
ax.set_ylim(-0.5, 1.5)
ax.legend()
ax.grid(True, alpha=0.3)
plt.show()

# TODO (bonus): construct explicit weights for a 2-2-1 network that computes XOR
# W1_xor = np.array(...)  # shape (2, 2)
# b1_xor = np.array(...)  # shape (2,)
# W2_xor = np.array(...)  # shape (2, 1)
# b2_xor = np.array(...)  # shape (1,)
#
# Z1 = X_xor_exact @ W1_xor + b1_xor
# H = sigmoid(Z1)
# Z2 = H @ W2_xor + b2_xor
# out = sigmoid(Z2)
# preds = (out.ravel() > 0.5).astype(int)
# assert np.array_equal(preds, y_xor_exact), f'XOR failed: {preds}'
# print(f'Hand-crafted XOR network outputs: {out.ravel()}')
# print(f'Predictions: {preds} == {y_xor_exact} ✓')

No description has been provided for this image

Exercise 4 — Coding: Train a 2-Layer MLP on XOR¶

Implement a minimal 2-layer MLP (2 inputs → $h$ hidden with tanh → 1 output with sigmoid) and train it on the exact XOR truth table using gradient descent.

Specifications:

Input: $X \in \mathbb{R}^{4 \times 2}$ (the 4 XOR points)
Target: $y \in \{0, 1\}^4$
Loss: Binary cross-entropy $L = -\frac{1}{n}\sum_i [y_i \log \hat{y}_i + (1-y_i)\log(1-\hat{y}_i)]$
Hidden units: $h = 4$
Learning rate: 1.0
Iterations: 5000

Tasks:

Implement forward_mlp and backward_mlp functions.
Run gradient descent and plot loss vs. iteration.
Print final predictions — they should round to $[0, 1, 1, 0]$.

Deterministic check: Final loss must be < 0.01.

In [6]:

Copied!





# XOR data
X_xor = np.array([[0, 0], [0, 1], [1, 0], [1, 1]], dtype=float)
y_xor = np.array([0, 1, 1, 0], dtype=float)

# Initialize parameters
rng_ex4 = np.random.default_rng(42)
h_units = 4
W1 = rng_ex4.normal(0, 1.0, size=(2, h_units))
b1 = np.zeros(h_units)
W2 = rng_ex4.normal(0, 1.0, size=(h_units, 1))
b2 = np.zeros(1)

lr = 1.0
n_steps = 5000


def forward_mlp(X, W1, b1, W2, b2):
    """Forward pass: X -> tanh hidden -> sigmoid output.

    Returns:
        y_hat: predictions shape (n, 1)
        cache: tuple of intermediates for backward pass
    """
    # TODO: implement
    # Z1 = ...
    # H = ...
    # Z2 = ...
    # y_hat = ...
    # return y_hat, (X, Z1, H, Z2, y_hat)
    pass


def backward_mlp(cache, y, W2):
    """Backward pass: compute gradients for W1, b1, W2, b2.

    Returns:
        dict with keys 'dW1', 'db1', 'dW2', 'db2'
    """
    # TODO: implement
    # X, Z1, H, Z2, y_hat = cache
    # n = X.shape[0]
    # dZ2 = (y_hat - y.reshape(-1, 1)) / n
    # dW2 = ...
    # db2 = ...
    # dH = ...
    # dZ1 = dH * (1 - H ** 2)  # tanh derivative
    # dW1 = ...
    # db1 = ...
    # return {'dW1': dW1, 'db1': db1, 'dW2': dW2, 'db2': db2}
    pass


def bce_loss(y_hat, y):
    """Binary cross-entropy loss."""
    eps = 1e-12
    y = y.reshape(-1, 1)
    return -np.mean(y * np.log(y_hat + eps) + (1 - y) * np.log(1 - y_hat + eps))


# TODO: training loop
# losses = []
# for step in range(n_steps):
#     y_hat, cache = forward_mlp(X_xor, W1, b1, W2, b2)
#     loss = bce_loss(y_hat, y_xor)
#     losses.append(loss)
#     grads = backward_mlp(cache, y_xor, W2)
#     W1 -= lr * grads['dW1']
#     b1 -= lr * grads['db1']
#     W2 -= lr * grads['dW2']
#     b2 -= lr * grads['db2']
#
# final_loss = losses[-1]
# y_final, _ = forward_mlp(X_xor, W1, b1, W2, b2)
# preds = (y_final.ravel() > 0.5).astype(int)
#
# print(f'Final loss: {final_loss:.6f}')
# print(f'Predictions: {y_final.ravel()}')
# print(f'Rounded:     {preds}')
# print(f'Target:      {y_xor.astype(int)}')
#
# assert final_loss < 0.01, f'Loss too high: {final_loss:.4f}'
# assert np.array_equal(preds, y_xor.astype(int)), f'XOR not solved: {preds}'
# print('\nXOR solved with loss < 0.01. ✓')
#
# fig, ax = plt.subplots(figsize=(7, 4))
# ax.plot(losses, color='steelblue')
# ax.set_xlabel('iteration')
# ax.set_ylabel('BCE loss')
# ax.set_title(f'XOR training — final loss = {final_loss:.6f}')
# ax.set_yscale('log')
# plt.show()
# XOR data
X_xor = np.array([[0, 0], [0, 1], [1, 0], [1, 1]], dtype=float)
y_xor = np.array([0, 1, 1, 0], dtype=float)

# Initialize parameters
rng_ex4 = np.random.default_rng(42)
h_units = 4
W1 = rng_ex4.normal(0, 1.0, size=(2, h_units))
b1 = np.zeros(h_units)
W2 = rng_ex4.normal(0, 1.0, size=(h_units, 1))
b2 = np.zeros(1)

lr = 1.0
n_steps = 5000


def forward_mlp(X, W1, b1, W2, b2):
    """Forward pass: X -> tanh hidden -> sigmoid output.

    Returns:
        y_hat: predictions shape (n, 1)
        cache: tuple of intermediates for backward pass
    """
    # TODO: implement
    # Z1 = ...
    # H = ...
    # Z2 = ...
    # y_hat = ...
    # return y_hat, (X, Z1, H, Z2, y_hat)
    pass


def backward_mlp(cache, y, W2):
    """Backward pass: compute gradients for W1, b1, W2, b2.

    Returns:
        dict with keys 'dW1', 'db1', 'dW2', 'db2'
    """
    # TODO: implement
    # X, Z1, H, Z2, y_hat = cache
    # n = X.shape[0]
    # dZ2 = (y_hat - y.reshape(-1, 1)) / n
    # dW2 = ...
    # db2 = ...
    # dH = ...
    # dZ1 = dH * (1 - H ** 2)  # tanh derivative
    # dW1 = ...
    # db1 = ...
    # return {'dW1': dW1, 'db1': db1, 'dW2': dW2, 'db2': db2}
    pass


def bce_loss(y_hat, y):
    """Binary cross-entropy loss."""
    eps = 1e-12
    y = y.reshape(-1, 1)
    return -np.mean(y * np.log(y_hat + eps) + (1 - y) * np.log(1 - y_hat + eps))


# TODO: training loop
# losses = []
# for step in range(n_steps):
#     y_hat, cache = forward_mlp(X_xor, W1, b1, W2, b2)
#     loss = bce_loss(y_hat, y_xor)
#     losses.append(loss)
#     grads = backward_mlp(cache, y_xor, W2)
#     W1 -= lr * grads['dW1']
#     b1 -= lr * grads['db1']
#     W2 -= lr * grads['dW2']
#     b2 -= lr * grads['db2']
#
# final_loss = losses[-1]
# y_final, _ = forward_mlp(X_xor, W1, b1, W2, b2)
# preds = (y_final.ravel() > 0.5).astype(int)
#
# print(f'Final loss: {final_loss:.6f}')
# print(f'Predictions: {y_final.ravel()}')
# print(f'Rounded:     {preds}')
# print(f'Target:      {y_xor.astype(int)}')
#
# assert final_loss < 0.01, f'Loss too high: {final_loss:.4f}'
# assert np.array_equal(preds, y_xor.astype(int)), f'XOR not solved: {preds}'
# print('\nXOR solved with loss < 0.01. ✓')
#
# fig, ax = plt.subplots(figsize=(7, 4))
# ax.plot(losses, color='steelblue')
# ax.set_xlabel('iteration')
# ax.set_ylabel('BCE loss')
# ax.set_title(f'XOR training — final loss = {final_loss:.6f}')
# ax.set_yscale('log')
# plt.show()

Exercise 5 — Conceptual: Vanishing Gradients — Sigmoid vs ReLU¶

Consider a deep network with $L$ layers and no skip connections. The gradient of the loss w.r.t. the first layer's pre-activation $Z_1$ is:

$$\frac{\partial L}{\partial Z_1} = \frac{\partial L}{\partial Z_L} \cdot \prod_{\ell=2}^{L} \left( W_\ell^T \odot \phi'(Z_{\ell-1}) \right)$$

where $\phi'$ is the activation derivative applied element-wise.

Questions:

For sigmoid activation, $\phi'(z) = \sigma(z)(1 - \sigma(z))$. What is the maximum value of $\sigma'(z)$? If every layer multiplies the gradient by at most this factor, how does $\|\partial L/\partial Z_1\|$ scale with depth $L$?
For ReLU activation, $\phi'(z) = \mathbb{1}[z > 0]$. Why does ReLU avoid the vanishing problem for units with $z > 0$? What different problem can ReLU cause ("dying ReLU")?
Suppose you have a 10-layer sigmoid network. Estimate the worst-case gradient shrinkage factor (ignoring the weight matrices — just the activation derivatives). Compare with a 10-layer ReLU network.
How does weight initialization (Xavier vs He) interact with the vanishing gradient problem? Why is He init preferred for ReLU?

In [7]:

Copied!





# Numerical demonstration: gradient magnitude through layers
# Compare sigmoid vs ReLU in a chain of 10 layers

n_layers = 10
d = 8  # width of each layer

# Simulate signal magnitude through activation derivatives
z_sample = np.linspace(-2, 2, d)  # typical pre-activations

# Sigmoid derivative chain
sig_deriv = sigmoid(z_sample) * (1 - sigmoid(z_sample))
sig_factor = np.mean(sig_deriv)  # average derivative magnitude
sig_shrinkage = sig_factor ** (n_layers - 1)

# ReLU derivative chain
relu_deriv = (z_sample > 0).astype(float)
relu_factor = np.mean(relu_deriv)  # fraction of active units
relu_shrinkage = relu_factor ** (n_layers - 1)

print(f'Average sigmoid derivative: {sig_factor:.4f}')
print(f'Average ReLU derivative:    {relu_factor:.4f}')
print(f'\nAfter {n_layers} layers (activation derivatives only):')
print(f'  Sigmoid shrinkage: {sig_factor:.4f}^{n_layers-1} = {sig_shrinkage:.2e}')
print(f'  ReLU shrinkage:    {relu_factor:.4f}^{n_layers-1} = {relu_shrinkage:.2e}')
print(f'  Ratio (ReLU / sigmoid): {relu_shrinkage / sig_shrinkage:.0f}x')

# TODO: answer the questions above in a markdown cell or as comments
# Key insight: sigmoid max derivative is 0.25 at z=0, so the gradient
# shrinks by at least 0.25 per layer → 0.25^9 ≈ 3.8e-6 for 10 layers.
# ReLU passes gradient unchanged for active units (derivative = 1),
# but completely kills gradient for inactive units (derivative = 0).
# Numerical demonstration: gradient magnitude through layers
# Compare sigmoid vs ReLU in a chain of 10 layers

n_layers = 10
d = 8  # width of each layer

# Simulate signal magnitude through activation derivatives
z_sample = np.linspace(-2, 2, d)  # typical pre-activations

# Sigmoid derivative chain
sig_deriv = sigmoid(z_sample) * (1 - sigmoid(z_sample))
sig_factor = np.mean(sig_deriv)  # average derivative magnitude
sig_shrinkage = sig_factor ** (n_layers - 1)

# ReLU derivative chain
relu_deriv = (z_sample > 0).astype(float)
relu_factor = np.mean(relu_deriv)  # fraction of active units
relu_shrinkage = relu_factor ** (n_layers - 1)

print(f'Average sigmoid derivative: {sig_factor:.4f}')
print(f'Average ReLU derivative:    {relu_factor:.4f}')
print(f'\nAfter {n_layers} layers (activation derivatives only):')
print(f'  Sigmoid shrinkage: {sig_factor:.4f}^{n_layers-1} = {sig_shrinkage:.2e}')
print(f'  ReLU shrinkage:    {relu_factor:.4f}^{n_layers-1} = {relu_shrinkage:.2e}')
print(f'  Ratio (ReLU / sigmoid): {relu_shrinkage / sig_shrinkage:.0f}x')

# TODO: answer the questions above in a markdown cell or as comments
# Key insight: sigmoid max derivative is 0.25 at z=0, so the gradient
# shrinks by at least 0.25 per layer → 0.25^9 ≈ 3.8e-6 for 10 layers.
# ReLU passes gradient unchanged for active units (derivative = 1),
# but completely kills gradient for inactive units (derivative = 0).

Average sigmoid derivative: 0.1788
Average ReLU derivative:    0.5000

After 10 layers (activation derivatives only):
  Sigmoid shrinkage: 0.1788^9 = 1.86e-07
  ReLU shrinkage:    0.5000^9 = 1.95e-03
  Ratio (ReLU / sigmoid): 10473x

In [8]:

Copied!





# Visualize: gradient magnitude vs depth for both activations
layers_range = np.arange(1, 21)
sig_max_deriv = 0.25  # max of sigma'(z)
relu_avg_deriv = 0.5  # ~50% of units active on average

fig, ax = plt.subplots(figsize=(8, 5))
ax.semilogy(layers_range, sig_max_deriv ** (layers_range - 1),
            'o-', color='crimson', label='Sigmoid (best case: 0.25 per layer)')
ax.semilogy(layers_range, relu_avg_deriv ** (layers_range - 1),
            's-', color='steelblue', label='ReLU (avg case: 0.5 per layer)')
ax.axhline(1e-6, color='gray', ls='--', alpha=0.5, label='gradient ≈ 0 threshold')
ax.set_xlabel('Network depth (number of layers)')
ax.set_ylabel('Gradient magnitude factor')
ax.set_title('Vanishing Gradients: Sigmoid vs ReLU')
ax.legend()
ax.grid(True, alpha=0.3)
plt.tight_layout()
plt.show()
# Visualize: gradient magnitude vs depth for both activations
layers_range = np.arange(1, 21)
sig_max_deriv = 0.25  # max of sigma'(z)
relu_avg_deriv = 0.5  # ~50% of units active on average

fig, ax = plt.subplots(figsize=(8, 5))
ax.semilogy(layers_range, sig_max_deriv ** (layers_range - 1),
            'o-', color='crimson', label='Sigmoid (best case: 0.25 per layer)')
ax.semilogy(layers_range, relu_avg_deriv ** (layers_range - 1),
            's-', color='steelblue', label='ReLU (avg case: 0.5 per layer)')
ax.axhline(1e-6, color='gray', ls='--', alpha=0.5, label='gradient ≈ 0 threshold')
ax.set_xlabel('Network depth (number of layers)')
ax.set_ylabel('Gradient magnitude factor')
ax.set_title('Vanishing Gradients: Sigmoid vs ReLU')
ax.legend()
ax.grid(True, alpha=0.3)
plt.tight_layout()
plt.show()