14 CNN — Exercises¶

Test your understanding of convolution, pooling, parameter sharing, receptive fields, and translation equivariance.

Prerequisites. Read theory.md and work through first_principles.ipynb before attempting these.

In [1]:

Copied!





import random

import matplotlib.pyplot as plt
import numpy as np

%load_ext autoreload
%autoreload 2

SEED = 42
random.seed(SEED)
rng = np.random.default_rng(SEED)

plt.rcParams["figure.dpi"] = 90
import random

import matplotlib.pyplot as plt
import numpy as np

%load_ext autoreload
%autoreload 2

SEED = 42
random.seed(SEED)
rng = np.random.default_rng(SEED)

plt.rcParams["figure.dpi"] = 90

Exercise 1 — Hand Calculation: Convolution Output¶

Compute the convolution (cross-correlation) of a $4 \times 4$ input with a $3 \times 3$ kernel, valid padding, stride 1.

Input $X$:

$$X = \begin{bmatrix} 1 & 0 & 2 & 1 \\ 0 & 1 & 1 & 0 \\ 2 & 0 & 0 & 1 \\ 1 & 1 & 2 & 0 \end{bmatrix}$$

Kernel $K$:

$$K = \begin{bmatrix} 1 & 0 & -1 \\ 1 & 0 & -1 \\ 1 & 0 & -1 \end{bmatrix}$$

Tasks (derive by hand, then verify numerically):

What is the output shape? (Use formula: $H_{\text{out}} = H - k_h + 1$)
Compute each element of the output $Y$.

Step-by-step for $Y_{0,0}$:

$$Y_{0,0} = \sum_{m=0}^{2} \sum_{n=0}^{2} K_{m,n} \cdot X_{m,n}$$

$$= 1 \cdot 1 + 0 \cdot 0 + (-1) \cdot 2 + 1 \cdot 0 + 0 \cdot 1 + (-1) \cdot 1 + 1 \cdot 2 + 0 \cdot 0 + (-1) \cdot 0$$

$$= 1 - 2 - 1 + 2 = 0$$

Expected results:

	$j=0$	$j=1$
$i=0$	$0$	$-1$
$i=1$	$0$	$1$

This kernel detects vertical edges: the left column sums with $+1$ and the right column sums with $-1$. Uniform regions give 0, edges give nonzero values.

Task 3 — output shapes with stride and padding. Using the general formula

$$H_{\text{out}} = \left\lfloor \frac{H + 2p - k}{s} \right\rfloor + 1,$$

compute $H_{\text{out}}$ by hand for each configuration:

$H$	$k$	$p$	$s$	Expected $H_{\text{out}}$
32	5	2	1	32
32	5	2	2	16
7	3	0	2	3
4	3	0	1	2

In [2]:

Copied!





X = np.array([[1, 0, 2, 1],
              [0, 1, 1, 0],
              [2, 0, 0, 1],
              [1, 1, 2, 0]], dtype=float)

K = np.array([[ 1,  0, -1],
              [ 1,  0, -1],
              [ 1,  0, -1]], dtype=float)

# Hand-computed values (see the worked Y_00 above; repeat for the rest)
Y_expected = np.array([[0, -1],
                       [0, 1]], dtype=float)


def cross_correlate_valid(X, K):
    """Valid-mode 2D cross-correlation (stride 1, no padding)."""
    kh, kw = K.shape
    out_h = X.shape[0] - kh + 1
    out_w = X.shape[1] - kw + 1
    Y = np.zeros((out_h, out_w))
    for i in range(out_h):
        for j in range(out_w):
            Y[i, j] = np.sum(X[i:i+kh, j:j+kw] * K)
    return Y


Y_computed = cross_correlate_valid(X, K)
assert Y_computed.shape == (2, 2), f"Expected (2, 2), got {Y_computed.shape}"
assert np.allclose(Y_computed, Y_expected, atol=1e-10)
print(f"Output shape: {Y_computed.shape} (4 - 3 + 1 = 2 in each dimension)")
print(f"Output:\n{Y_computed}")
print("Hand calculation matches. ✓")


# Task 3: output-shape formula vs brute-force placement count
def conv_output_size(n, k, p=0, s=1):
    return (n + 2 * p - k) // s + 1


cases = [(32, 5, 2, 1, 32), (32, 5, 2, 2, 16), (7, 3, 0, 2, 3), (4, 3, 0, 1, 2)]
for n, k, p, s, expected in cases:
    formula = conv_output_size(n, k, p, s)
    # Brute force: count kernel placements i = 0, s, 2s, ... with i + k <= n + 2p
    brute = len(range(0, n + 2 * p - k + 1, s))
    assert formula == brute == expected, (n, k, p, s, formula, brute, expected)
    print(f"H={n:2d}, k={k}, p={p}, s={s} → H_out = {formula:2d} (formula = brute force) ✓")
print("All output-shape checks passed. ✓")
X = np.array([[1, 0, 2, 1],
              [0, 1, 1, 0],
              [2, 0, 0, 1],
              [1, 1, 2, 0]], dtype=float)

K = np.array([[ 1,  0, -1],
              [ 1,  0, -1],
              [ 1,  0, -1]], dtype=float)

# Hand-computed values (see the worked Y_00 above; repeat for the rest)
Y_expected = np.array([[0, -1],
                       [0, 1]], dtype=float)


def cross_correlate_valid(X, K):
    """Valid-mode 2D cross-correlation (stride 1, no padding)."""
    kh, kw = K.shape
    out_h = X.shape[0] - kh + 1
    out_w = X.shape[1] - kw + 1
    Y = np.zeros((out_h, out_w))
    for i in range(out_h):
        for j in range(out_w):
            Y[i, j] = np.sum(X[i:i+kh, j:j+kw] * K)
    return Y


Y_computed = cross_correlate_valid(X, K)
assert Y_computed.shape == (2, 2), f"Expected (2, 2), got {Y_computed.shape}"
assert np.allclose(Y_computed, Y_expected, atol=1e-10)
print(f"Output shape: {Y_computed.shape} (4 - 3 + 1 = 2 in each dimension)")
print(f"Output:\n{Y_computed}")
print("Hand calculation matches. ✓")


# Task 3: output-shape formula vs brute-force placement count
def conv_output_size(n, k, p=0, s=1):
    return (n + 2 * p - k) // s + 1


cases = [(32, 5, 2, 1, 32), (32, 5, 2, 2, 16), (7, 3, 0, 2, 3), (4, 3, 0, 1, 2)]
for n, k, p, s, expected in cases:
    formula = conv_output_size(n, k, p, s)
    # Brute force: count kernel placements i = 0, s, 2s, ... with i + k <= n + 2p
    brute = len(range(0, n + 2 * p - k + 1, s))
    assert formula == brute == expected, (n, k, p, s, formula, brute, expected)
    print(f"H={n:2d}, k={k}, p={p}, s={s} → H_out = {formula:2d} (formula = brute force) ✓")
print("All output-shape checks passed. ✓")

Output shape: (2, 2) (4 - 3 + 1 = 2 in each dimension)
Output:
[[ 0. -1.]
 [ 0.  1.]]
Hand calculation matches. ✓
H=32, k=5, p=2, s=1 → H_out = 32 (formula = brute force) ✓
H=32, k=5, p=2, s=2 → H_out = 16 (formula = brute force) ✓
H= 7, k=3, p=0, s=2 → H_out =  3 (formula = brute force) ✓
H= 4, k=3, p=0, s=1 → H_out =  2 (formula = brute force) ✓
All output-shape checks passed. ✓

Exercise 2 — Coding: Implement Cross-Correlation with Padding¶

Implement a cross-correlation function that supports both valid and same padding modes.

Specifications:

Input: 2D array $X$ of shape $(H, W)$ and kernel $K$ of shape $(k_h, k_w)$
mode='valid': no padding, output $(H - k_h + 1) \times (W - k_w + 1)$
mode='same': zero-pad so output has same size as input ($H \times W$), assuming stride 1 and odd kernel size
Stride is always 1

Deterministic check: Test on the given input/kernel pairs.

Hint for same padding: Pad by $\lfloor k_h / 2 \rfloor$ rows on top/bottom and $\lfloor k_w / 2 \rfloor$ columns on left/right, then apply valid-mode cross-correlation on the padded input.

In [3]:

Copied!





def cross_correlate(X, K, mode='valid'):
    """2D cross-correlation with valid or same padding.

    Args:
        X: input array, shape (H, W)
        K: kernel array, shape (kh, kw)
        mode: 'valid' or 'same'

    Returns:
        Y: output array
    """
    # TODO: implement
    # For 'same' mode: pad X with zeros, then do valid correlation
    # pad_h = K.shape[0] // 2
    # pad_w = K.shape[1] // 2
    # X_padded = np.pad(X, ((pad_h, pad_h), (pad_w, pad_w)), mode='constant')
    pass
def cross_correlate(X, K, mode='valid'):
    """2D cross-correlation with valid or same padding.

    Args:
        X: input array, shape (H, W)
        K: kernel array, shape (kh, kw)
        mode: 'valid' or 'same'

    Returns:
        Y: output array
    """
    # TODO: implement
    # For 'same' mode: pad X with zeros, then do valid correlation
    # pad_h = K.shape[0] // 2
    # pad_w = K.shape[1] // 2
    # X_padded = np.pad(X, ((pad_h, pad_h), (pad_w, pad_w)), mode='constant')
    pass

Solution 2¶

In [4]:

Copied!





def cross_correlate(X, K, mode='valid'):
    """2D cross-correlation with valid or same padding (stride 1).

    Args:
        X: input array, shape (H, W)
        K: kernel array, shape (kh, kw)
        mode: 'valid' or 'same'

    Returns:
        Y: output array
    """
    if mode == 'same':
        pad_h = K.shape[0] // 2
        pad_w = K.shape[1] // 2
        X = np.pad(X, ((pad_h, pad_h), (pad_w, pad_w)), mode='constant')
    elif mode != 'valid':
        raise ValueError(f"Unknown mode: {mode!r}")
    kh, kw = K.shape
    out_h = X.shape[0] - kh + 1
    out_w = X.shape[1] - kw + 1
    Y = np.zeros((out_h, out_w))
    for i in range(out_h):
        for j in range(out_w):
            Y[i, j] = np.sum(X[i:i+kh, j:j+kw] * K)
    return Y


# Test cases
X_test = np.array([[1, 2, 3, 4],
                   [5, 6, 7, 8],
                   [9, 10, 11, 12],
                   [1, 2, 3, 4]], dtype=float)

K_test = np.array([[1, 0],
                   [0, -1]], dtype=float)

K_laplacian = np.array([[0, 1, 0],
                        [1, -4, 1],
                        [0, 1, 0]], dtype=float)

# Valid mode: Y[i,j] = X[i,j] - X[i+1,j+1] → -5 in rows 0-1, +7 in row 2
Y_valid = cross_correlate(X_test, K_test, mode='valid')
Y_valid_expected = np.array([[-5, -5, -5],
                             [-5, -5, -5],
                             [7, 7, 7]], dtype=float)
assert Y_valid.shape == (3, 3), f"Expected (3, 3), got {Y_valid.shape}"
assert np.allclose(Y_valid, Y_valid_expected, atol=1e-10)
print(f"Valid mode output shape: {Y_valid.shape} ✓")
print(f"Valid mode output:\n{Y_valid}")

# Same mode: output keeps the input size; its interior must equal the valid output
Y_same = cross_correlate(X_test, K_laplacian, mode='same')
assert Y_same.shape == (4, 4), f"Expected (4, 4), got {Y_same.shape}"
Y_valid_lap = cross_correlate(X_test, K_laplacian, mode='valid')
assert np.allclose(Y_same[1:-1, 1:-1], Y_valid_lap, atol=1e-10), (
    "Interior of same-mode output must match valid-mode output"
)
# Hand check of the interior: Laplacian of the linear rows 0-2 is 0
assert np.allclose(Y_valid_lap, [[0, 0], [-12, -12]], atol=1e-10)
print(f"\nSame mode output shape: {Y_same.shape} ✓")
print(f"Same mode output:\n{Y_same}")
print("\nAll cross-correlation tests passed. ✓")
def cross_correlate(X, K, mode='valid'):
    """2D cross-correlation with valid or same padding (stride 1).

    Args:
        X: input array, shape (H, W)
        K: kernel array, shape (kh, kw)
        mode: 'valid' or 'same'

    Returns:
        Y: output array
    """
    if mode == 'same':
        pad_h = K.shape[0] // 2
        pad_w = K.shape[1] // 2
        X = np.pad(X, ((pad_h, pad_h), (pad_w, pad_w)), mode='constant')
    elif mode != 'valid':
        raise ValueError(f"Unknown mode: {mode!r}")
    kh, kw = K.shape
    out_h = X.shape[0] - kh + 1
    out_w = X.shape[1] - kw + 1
    Y = np.zeros((out_h, out_w))
    for i in range(out_h):
        for j in range(out_w):
            Y[i, j] = np.sum(X[i:i+kh, j:j+kw] * K)
    return Y


# Test cases
X_test = np.array([[1, 2, 3, 4],
                   [5, 6, 7, 8],
                   [9, 10, 11, 12],
                   [1, 2, 3, 4]], dtype=float)

K_test = np.array([[1, 0],
                   [0, -1]], dtype=float)

K_laplacian = np.array([[0, 1, 0],
                        [1, -4, 1],
                        [0, 1, 0]], dtype=float)

# Valid mode: Y[i,j] = X[i,j] - X[i+1,j+1] → -5 in rows 0-1, +7 in row 2
Y_valid = cross_correlate(X_test, K_test, mode='valid')
Y_valid_expected = np.array([[-5, -5, -5],
                             [-5, -5, -5],
                             [7, 7, 7]], dtype=float)
assert Y_valid.shape == (3, 3), f"Expected (3, 3), got {Y_valid.shape}"
assert np.allclose(Y_valid, Y_valid_expected, atol=1e-10)
print(f"Valid mode output shape: {Y_valid.shape} ✓")
print(f"Valid mode output:\n{Y_valid}")

# Same mode: output keeps the input size; its interior must equal the valid output
Y_same = cross_correlate(X_test, K_laplacian, mode='same')
assert Y_same.shape == (4, 4), f"Expected (4, 4), got {Y_same.shape}"
Y_valid_lap = cross_correlate(X_test, K_laplacian, mode='valid')
assert np.allclose(Y_same[1:-1, 1:-1], Y_valid_lap, atol=1e-10), (
    "Interior of same-mode output must match valid-mode output"
)
# Hand check of the interior: Laplacian of the linear rows 0-2 is 0
assert np.allclose(Y_valid_lap, [[0, 0], [-12, -12]], atol=1e-10)
print(f"\nSame mode output shape: {Y_same.shape} ✓")
print(f"Same mode output:\n{Y_same}")
print("\nAll cross-correlation tests passed. ✓")

Valid mode output shape: (3, 3) ✓
Valid mode output:
[[-5. -5. -5.]
 [-5. -5. -5.]
 [ 7.  7.  7.]]

Same mode output shape: (4, 4) ✓
Same mode output:
[[  3.   2.   1.  -5.]
 [ -4.   0.   0.  -9.]
 [-20. -12. -12. -25.]
 [  7.   6.   5.  -1.]]

All cross-correlation tests passed. ✓

Exercise 3 — Coding: Max Pooling Forward and Backward¶

Implement max pooling on a single-channel 2D input and its backward pass, then verify the backward pass with central finite differences.

Forward (window $p \times p$, stride $p$, non-overlapping; assume $H$ and $W$ are divisible by $p$):

$$Y_{i,j} = \max_{0 \le m < p,\; 0 \le n < p} X_{ip + m,\; jp + n}.$$

Backward. The gradient flows only to the argmax position of each window:

$$\frac{\partial L}{\partial X_{a,b}} = \begin{cases} \left(\dfrac{\partial L}{\partial Y}\right)_{i,j} & \text{if } (a, b) \text{ is the argmax of window } (i, j) \\ 0 & \text{otherwise.} \end{cases}$$

Specifications:

max_pool(X, p) returns (Y, mask) where mask has a 1 at each window's argmax position and 0 elsewhere.
max_pool_backward(dY, mask, p) returns dX with the same shape as X.

Deterministic check. For the scalar loss $L(X) = \sum_{i,j} W_{i,j} \, Y_{i,j}$ with a fixed random $W$, the upstream gradient is $\partial L / \partial Y = W$. Compare your analytic dX against the central difference $\left(L(X + \varepsilon e_{ab}) - L(X - \varepsilon e_{ab})\right) / (2\varepsilon)$ for every entry $(a, b)$, with np.allclose(..., atol=1e-6).

(The first-principles notebook builds a batched MaxPool2D layer; here you re-derive the core 2D operation and, unlike there, verify the backward pass numerically.)

In [5]:

Copied!





def max_pool(X, p):
    """Non-overlapping p x p max pooling of a 2D array.

    Args:
        X: input array, shape (H, W), H and W divisible by p
        p: pooling window size (stride = p)

    Returns:
        Y: pooled output, shape (H // p, W // p)
        mask: same shape as X, 1.0 at each window's argmax, 0.0 elsewhere
    """
    # TODO: implement
    # Hint: np.unravel_index(np.argmax(window), (p, p)) locates the max.
    pass


def max_pool_backward(dY, mask, p):
    """Backward pass: route each upstream gradient to its argmax position.

    Args:
        dY: upstream gradient, shape (H // p, W // p)
        mask: argmax mask from max_pool
        p: pooling window size

    Returns:
        dX: gradient w.r.t. X, same shape as mask
    """
    # TODO: implement
    # Hint: upsample dY with np.repeat along both axes, then apply the mask.
    pass
def max_pool(X, p):
    """Non-overlapping p x p max pooling of a 2D array.

    Args:
        X: input array, shape (H, W), H and W divisible by p
        p: pooling window size (stride = p)

    Returns:
        Y: pooled output, shape (H // p, W // p)
        mask: same shape as X, 1.0 at each window's argmax, 0.0 elsewhere
    """
    # TODO: implement
    # Hint: np.unravel_index(np.argmax(window), (p, p)) locates the max.
    pass


def max_pool_backward(dY, mask, p):
    """Backward pass: route each upstream gradient to its argmax position.

    Args:
        dY: upstream gradient, shape (H // p, W // p)
        mask: argmax mask from max_pool
        p: pooling window size

    Returns:
        dX: gradient w.r.t. X, same shape as mask
    """
    # TODO: implement
    # Hint: upsample dY with np.repeat along both axes, then apply the mask.
    pass

Solution 3¶

In [6]:

Copied!





def max_pool(X, p):
    """Non-overlapping p x p max pooling of a 2D array."""
    H, W = X.shape
    out_h, out_w = H // p, W // p
    Y = np.zeros((out_h, out_w))
    mask = np.zeros_like(X)
    for i in range(out_h):
        for j in range(out_w):
            window = X[i*p:(i+1)*p, j*p:(j+1)*p]
            Y[i, j] = np.max(window)
            m, n = np.unravel_index(np.argmax(window), (p, p))
            mask[i*p + m, j*p + n] = 1.0
    return Y, mask


def max_pool_backward(dY, mask, p):
    """Backward pass: route each upstream gradient to its argmax position."""
    return mask * np.repeat(np.repeat(dY, p, axis=0), p, axis=1)


# Forward check on a hand-computed example
X_demo = np.array([[2, 1, 0, 3],
                   [0, 4, 5, 1],
                   [7, 2, 1, 1],
                   [3, 0, 2, 6]], dtype=float)
Y_demo, mask_demo = max_pool(X_demo, 2)
assert np.allclose(Y_demo, [[4, 5], [7, 6]], atol=1e-12)
assert mask_demo.sum() == 4  # exactly one max per window
print(f"Forward: {X_demo.shape} → {Y_demo.shape}, maxes = {Y_demo.ravel()} ✓")

# Backward check via central finite differences on L(X) = sum(W_loss * Y)
X_rand = rng.normal(size=(6, 6))
W_loss = rng.normal(size=(3, 3))


def loss(X):
    Y, _ = max_pool(X, 2)
    return np.sum(W_loss * Y)


Y_rand, mask_rand = max_pool(X_rand, 2)
dX_analytic = max_pool_backward(W_loss, mask_rand, 2)

eps = 1e-6
dX_numeric = np.zeros_like(X_rand)
for a in range(X_rand.shape[0]):
    for b in range(X_rand.shape[1]):
        X_plus, X_minus = X_rand.copy(), X_rand.copy()
        X_plus[a, b] += eps
        X_minus[a, b] -= eps
        dX_numeric[a, b] = (loss(X_plus) - loss(X_minus)) / (2 * eps)

max_err = np.max(np.abs(dX_analytic - dX_numeric))
assert np.allclose(dX_analytic, dX_numeric, atol=1e-6), f"max error {max_err:.2e}"
print(f"Backward: max |analytic - numeric| = {max_err:.2e}")
print("Finite-difference gradient check passed. ✓")
def max_pool(X, p):
    """Non-overlapping p x p max pooling of a 2D array."""
    H, W = X.shape
    out_h, out_w = H // p, W // p
    Y = np.zeros((out_h, out_w))
    mask = np.zeros_like(X)
    for i in range(out_h):
        for j in range(out_w):
            window = X[i*p:(i+1)*p, j*p:(j+1)*p]
            Y[i, j] = np.max(window)
            m, n = np.unravel_index(np.argmax(window), (p, p))
            mask[i*p + m, j*p + n] = 1.0
    return Y, mask


def max_pool_backward(dY, mask, p):
    """Backward pass: route each upstream gradient to its argmax position."""
    return mask * np.repeat(np.repeat(dY, p, axis=0), p, axis=1)


# Forward check on a hand-computed example
X_demo = np.array([[2, 1, 0, 3],
                   [0, 4, 5, 1],
                   [7, 2, 1, 1],
                   [3, 0, 2, 6]], dtype=float)
Y_demo, mask_demo = max_pool(X_demo, 2)
assert np.allclose(Y_demo, [[4, 5], [7, 6]], atol=1e-12)
assert mask_demo.sum() == 4  # exactly one max per window
print(f"Forward: {X_demo.shape} → {Y_demo.shape}, maxes = {Y_demo.ravel()} ✓")

# Backward check via central finite differences on L(X) = sum(W_loss * Y)
X_rand = rng.normal(size=(6, 6))
W_loss = rng.normal(size=(3, 3))


def loss(X):
    Y, _ = max_pool(X, 2)
    return np.sum(W_loss * Y)


Y_rand, mask_rand = max_pool(X_rand, 2)
dX_analytic = max_pool_backward(W_loss, mask_rand, 2)

eps = 1e-6
dX_numeric = np.zeros_like(X_rand)
for a in range(X_rand.shape[0]):
    for b in range(X_rand.shape[1]):
        X_plus, X_minus = X_rand.copy(), X_rand.copy()
        X_plus[a, b] += eps
        X_minus[a, b] -= eps
        dX_numeric[a, b] = (loss(X_plus) - loss(X_minus)) / (2 * eps)

max_err = np.max(np.abs(dX_analytic - dX_numeric))
assert np.allclose(dX_analytic, dX_numeric, atol=1e-6), f"max error {max_err:.2e}"
print(f"Backward: max |analytic - numeric| = {max_err:.2e}")
print("Finite-difference gradient check passed. ✓")

Forward: (4, 4) → (2, 2), maxes = [4. 5. 7. 6.] ✓
Backward: max |analytic - numeric| = 1.05e-10
Finite-difference gradient check passed. ✓

Questions:

A dense layer connecting a $32 \times 32$ grayscale image to 64 hidden units needs how many parameters? A convolutional layer with 64 filters of size $5 \times 5$ needs how many? Compute both and find the ratio.
Why is parameter sharing a good inductive bias for images? What assumption about the data does it encode?
Give an example of data where parameter sharing is harmful (i.e., the CNN's assumption is violated).

Part B: Receptive field of a 2-layer CNN¶

Questions:

Consider two stacked $3 \times 3$ conv layers (stride 1, no padding). What is the receptive field of a single unit in the output of the second layer? Use the formula: $r = L(k-1) + 1$.
How many parameters does this 2-layer stack have (single input channel, single output channel per layer, ignoring biases)? Compare with a single $5 \times 5$ filter that achieves the same receptive field.
Why do modern architectures (VGG, ResNet) prefer stacks of $3 \times 3$ filters over larger filters?

In [7]:

Copied!





# Part A: parameter count computation
dense_params = 32 * 32 * 64 + 64   # weights + bias
conv_params = 64 * 1 * 5 * 5 + 64  # filters + bias
ratio = dense_params / conv_params
assert dense_params == 65600 and conv_params == 1664
print(f"Dense parameters: {dense_params:,}")
print(f"Conv parameters:  {conv_params:,}")
print(f"Ratio: {ratio:.1f}×")
print()

# Part B: receptive field of L stacked k x k layers (stride 1): r = L(k-1) + 1
n_layers, k = 2, 3
r = n_layers * (k - 1) + 1
assert r == 5

# Parameters, single channel per layer, no bias
two_3x3 = 2 * (3 * 3)  # = 18
one_5x5 = 5 * 5        # = 25
print(f"Receptive field of two 3×3 layers: {r}×{r}")
print(f"Parameters — two 3×3: {two_3x3}, one 5×5: {one_5x5}")
print(f"Two 3×3 layers use {one_5x5 - two_3x3} fewer parameters")
print("Plus: two 3×3 layers have two nonlinearities vs one.")
# Part A: parameter count computation
dense_params = 32 * 32 * 64 + 64   # weights + bias
conv_params = 64 * 1 * 5 * 5 + 64  # filters + bias
ratio = dense_params / conv_params
assert dense_params == 65600 and conv_params == 1664
print(f"Dense parameters: {dense_params:,}")
print(f"Conv parameters:  {conv_params:,}")
print(f"Ratio: {ratio:.1f}×")
print()

# Part B: receptive field of L stacked k x k layers (stride 1): r = L(k-1) + 1
n_layers, k = 2, 3
r = n_layers * (k - 1) + 1
assert r == 5

# Parameters, single channel per layer, no bias
two_3x3 = 2 * (3 * 3)  # = 18
one_5x5 = 5 * 5        # = 25
print(f"Receptive field of two 3×3 layers: {r}×{r}")
print(f"Parameters — two 3×3: {two_3x3}, one 5×5: {one_5x5}")
print(f"Two 3×3 layers use {one_5x5 - two_3x3} fewer parameters")
print("Plus: two 3×3 layers have two nonlinearities vs one.")

Dense parameters: 65,600
Conv parameters:  1,664
Ratio: 39.4×

Receptive field of two 3×3 layers: 5×5
Parameters — two 3×3: 18, one 5×5: 25
Two 3×3 layers use 7 fewer parameters
Plus: two 3×3 layers have two nonlinearities vs one.

Expected answers (reveal after attempting)¶

Click to reveal

Part A:

Dense: $32 \times 32 \times 64 + 64 = 65{,}600$ parameters. Conv: $64 \times 1 \times 5 \times 5 + 64 = 1{,}664$ parameters. Ratio: $\approx 39.4\times$.
Parameter sharing encodes the assumption that the same local pattern (edge, texture, corner) is useful regardless of where it appears in the image. This is called translation equivariance.
Shuffled-pixel images, or data where position carries unique meaning (e.g., a form where field 1 is always top-left and field 2 is always bottom-right). Here, sharing weights across positions hurts because different positions require different processing.

Part B:

$r = 2 \times (3-1) + 1 = 5$. Each output unit sees a $5 \times 5$ region of the original input.
Two $3 \times 3$ layers: $2 \times 9 = 18$ parameters. One $5 \times 5$ layer: $25$ parameters. The stack is cheaper.
Three advantages: (a) fewer parameters for the same receptive field, (b) more nonlinear activations (one between each layer), increasing the model's representational power, and (c) easier to train with modern optimizers.

Exercise 5 — Failure Analysis: Stride and Pooling Break Translation Equivariance¶

Theory §7 states that stride-1 convolution is equivariant to translation: shifting the input shifts every feature map by the same amount. Downsampling (stride $s > 1$ or pooling) samples the feature map on a coarser grid, and that sampling step does not commute with arbitrary shifts.

Tasks (verify each claim numerically below):

Embed a fixed random $3 \times 3$ pattern in a zero image at column $c$ and at column $c + 1$. Show that the stride-1 valid convolution outputs are exact shifted copies of each other.
Subsample both outputs with stride 2 (keep every second row and column). Show that the two subsampled outputs are not shifted copies of each other — a 1-pixel input shift changes the values, not just their position.
Shift the pattern by 2 pixels (a multiple of the stride) instead. Show that equivariance is restored: the subsampled output shifts by exactly 1 cell. Confirm the same behavior for $2 \times 2$ max pooling.

Questions:

Why does a shift by 1 pixel break stride-2 equivariance while a shift by 2 pixels does not?
A classifier built as conv → pool → ... → global average pooling is approximately shift-invariant, yet its logits can still jitter under 1-pixel input shifts. Which layers cause the jitter?
What practical remedies reduce this sensitivity?

Solution 5¶

Subsampling keeps positions $0, 2, 4, \ldots$ of the feature map. A 1-pixel input shift moves the feature response onto the odd positions, which the stride-2 grid never reads — the surviving samples are taken from different points of the response, so the values themselves change (aliasing). A 2-pixel shift moves the response by a whole grid cell, so the same values reappear one cell over.
Every layer that downsamples — strided convolutions and pooling layers — is only equivariant to shifts that are multiples of its stride. Composing them leaves the network exactly equivariant only to shifts that are multiples of the product of all strides (e.g. 32 pixels for a typical ResNet), so almost all 1-pixel shifts perturb the features feeding the final pooling.
Blur (low-pass) before downsampling — anti-aliased pooling — plus shift augmentation during training; or reduce the amount of downsampling.

In [8]:

Copied!





def embed(pattern, H, W, row, col):
    """Place a small pattern inside a zero image at (row, col)."""
    X = np.zeros((H, W))
    X[row:row + pattern.shape[0], col:col + pattern.shape[1]] = pattern
    return X


pattern = rng.normal(size=(3, 3))
K_rand = rng.normal(size=(3, 3))

X_base = embed(pattern, 12, 12, 4, 3)
X_shift1 = embed(pattern, 12, 12, 4, 4)  # shifted right by 1 pixel
X_shift2 = embed(pattern, 12, 12, 4, 5)  # shifted right by 2 pixels

# Reuses cross_correlate from Solution 2 (valid mode, stride 1) → 10x10 maps
Y_base = cross_correlate(X_base, K_rand, mode='valid')
Y_shift1 = cross_correlate(X_shift1, K_rand, mode='valid')
Y_shift2 = cross_correlate(X_shift2, K_rand, mode='valid')

# Claim 1: stride-1 convolution is translation-equivariant
assert np.allclose(Y_shift1[:, 1:], Y_base[:, :-1], atol=1e-12)
print("Claim 1: stride-1 conv output is an exact 1-column shifted copy ✓")

# Claim 2: stride-2 subsampling breaks equivariance for a 1-pixel shift
D_base = Y_base[::2, ::2]
D_shift1 = Y_shift1[::2, ::2]
same_place = np.allclose(D_shift1, D_base, atol=1e-8)
shifted_copy = np.allclose(D_shift1[:, 1:], D_base[:, :-1], atol=1e-8)
assert not same_place and not shifted_copy
print("Claim 2: after stride-2 subsampling the shifted output is neither")
print("         equal to nor a shifted copy of the original ✓")
print(f"         max |D_shift1 - D_base| = {np.max(np.abs(D_shift1 - D_base)):.3f}")

# Claim 3: a shift by the stride (2 px) survives — output shifts by 1 cell
D_shift2 = Y_shift2[::2, ::2]
assert np.allclose(D_shift2[:, 1:], D_base[:, :-1], atol=1e-12)

# Same story for 2x2 max pooling (uses max_pool from Solution 3)
P_base, _ = max_pool(Y_base, 2)
P_shift1, _ = max_pool(Y_shift1, 2)
P_shift2, _ = max_pool(Y_shift2, 2)
assert not np.allclose(P_shift1[:, 1:], P_base[:, :-1], atol=1e-8)
assert np.allclose(P_shift2[:, 1:], P_base[:, :-1], atol=1e-12)
print("Claim 3: shifting by a multiple of the stride (2 px) shifts the")
print("         subsampled and max-pooled outputs by exactly 1 cell ✓")
print("All equivariance checks passed. ✓")
def embed(pattern, H, W, row, col):
    """Place a small pattern inside a zero image at (row, col)."""
    X = np.zeros((H, W))
    X[row:row + pattern.shape[0], col:col + pattern.shape[1]] = pattern
    return X


pattern = rng.normal(size=(3, 3))
K_rand = rng.normal(size=(3, 3))

X_base = embed(pattern, 12, 12, 4, 3)
X_shift1 = embed(pattern, 12, 12, 4, 4)  # shifted right by 1 pixel
X_shift2 = embed(pattern, 12, 12, 4, 5)  # shifted right by 2 pixels

# Reuses cross_correlate from Solution 2 (valid mode, stride 1) → 10x10 maps
Y_base = cross_correlate(X_base, K_rand, mode='valid')
Y_shift1 = cross_correlate(X_shift1, K_rand, mode='valid')
Y_shift2 = cross_correlate(X_shift2, K_rand, mode='valid')

# Claim 1: stride-1 convolution is translation-equivariant
assert np.allclose(Y_shift1[:, 1:], Y_base[:, :-1], atol=1e-12)
print("Claim 1: stride-1 conv output is an exact 1-column shifted copy ✓")

# Claim 2: stride-2 subsampling breaks equivariance for a 1-pixel shift
D_base = Y_base[::2, ::2]
D_shift1 = Y_shift1[::2, ::2]
same_place = np.allclose(D_shift1, D_base, atol=1e-8)
shifted_copy = np.allclose(D_shift1[:, 1:], D_base[:, :-1], atol=1e-8)
assert not same_place and not shifted_copy
print("Claim 2: after stride-2 subsampling the shifted output is neither")
print("         equal to nor a shifted copy of the original ✓")
print(f"         max |D_shift1 - D_base| = {np.max(np.abs(D_shift1 - D_base)):.3f}")

# Claim 3: a shift by the stride (2 px) survives — output shifts by 1 cell
D_shift2 = Y_shift2[::2, ::2]
assert np.allclose(D_shift2[:, 1:], D_base[:, :-1], atol=1e-12)

# Same story for 2x2 max pooling (uses max_pool from Solution 3)
P_base, _ = max_pool(Y_base, 2)
P_shift1, _ = max_pool(Y_shift1, 2)
P_shift2, _ = max_pool(Y_shift2, 2)
assert not np.allclose(P_shift1[:, 1:], P_base[:, :-1], atol=1e-8)
assert np.allclose(P_shift2[:, 1:], P_base[:, :-1], atol=1e-12)
print("Claim 3: shifting by a multiple of the stride (2 px) shifts the")
print("         subsampled and max-pooled outputs by exactly 1 cell ✓")
print("All equivariance checks passed. ✓")

Claim 1: stride-1 conv output is an exact 1-column shifted copy ✓
Claim 2: after stride-2 subsampling the shifted output is neither
         equal to nor a shifted copy of the original ✓
         max |D_shift1 - D_base| = 2.050
Claim 3: shifting by a multiple of the stride (2 px) shifts the
         subsampled and max-pooled outputs by exactly 1 cell ✓
All equivariance checks passed. ✓

14 CNN — Exercises¶

Exercise 1 — Hand Calculation: Convolution Output¶

Exercise 2 — Coding: Implement Cross-Correlation with Padding¶

Solution 2¶

Exercise 3 — Coding: Max Pooling Forward and Backward¶

Solution 3¶

Exercise 4 — Conceptual: Parameter Sharing and Receptive Field¶

Part A: Why does parameter sharing make CNNs efficient?¶

Part B: Receptive field of a 2-layer CNN¶

Expected answers (reveal after attempting)¶

Exercise 5 — Failure Analysis: Stride and Pooling Break Translation Equivariance¶

Solution 5¶