15 RNN / LSTM — Exercises¶
Test your understanding of recurrent computation, BPTT, and gating mechanisms.
Prerequisites. Read theory.md and work through first_principles.ipynb before attempting these.
import random
import matplotlib.pyplot as plt
import numpy as np
%load_ext autoreload
%autoreload 2
SEED = 42
random.seed(SEED)
rng = np.random.default_rng(SEED)
Exercise 1 — Hand Calculation: One RNN Forward Step¶
Consider a vanilla RNN with $d = 2$ (input dim) and $m = 2$ (hidden dim).
Recurrence: $h_t = \tanh(W_{xh}\, x_t + W_{hh}\, h_{t-1} + b_h)$
Given:
| Parameter | Value |
|---|---|
| $W_{xh}$ | $\begin{bmatrix} 0.5 & 0.3 \\\\ -0.2 & 0.4 \end{bmatrix}$ |
| $W_{hh}$ | $\begin{bmatrix} 0.1 & -0.1 \\\\ 0.2 & 0.3 \end{bmatrix}$ |
| $b_h$ | $\begin{bmatrix} 0.0 \\\\ 0.1 \end{bmatrix}$ |
| $h_0$ | $\begin{bmatrix} 0.0 \\\\ 0.0 \end{bmatrix}$ |
| $x_1$ | $\begin{bmatrix} 1.0 \\\\ 0.5 \end{bmatrix}$ |
| $x_2$ | $\begin{bmatrix} -0.5 \\\\ 1.0 \end{bmatrix}$ |
Tasks (derive by hand, then verify numerically):
- Compute $z_1 = W_{xh}\, x_1 + W_{hh}\, h_0 + b_h$ and $h_1 = \tanh(z_1)$.
- Compute $z_2 = W_{xh}\, x_2 + W_{hh}\, h_1 + b_h$ and $h_2 = \tanh(z_2)$.
Step-by-step for $h_1$:
$$W_{xh}\, x_1 = \begin{bmatrix} 0.5 \cdot 1.0 + 0.3 \cdot 0.5 \\ -0.2 \cdot 1.0 + 0.4 \cdot 0.5 \end{bmatrix} = \begin{bmatrix} 0.65 \\ 0.0 \end{bmatrix}$$
$$W_{hh}\, h_0 = \begin{bmatrix} 0.0 \\\\ 0.0 \end{bmatrix}$$
$$z_1 = \begin{bmatrix} 0.65 \\ 0.0 \end{bmatrix} + \begin{bmatrix} 0.0 \\\\ 0.0 \end{bmatrix} + \begin{bmatrix} 0.0 \\\\ 0.1 \end{bmatrix} = \begin{bmatrix} 0.65 \\ 0.1 \end{bmatrix}$$
$$h_1 = \tanh(z_1) = \begin{bmatrix} \tanh(0.65) \\ \tanh(0.1) \end{bmatrix} \approx \begin{bmatrix} 0.5717 \\ 0.0997 \end{bmatrix}$$
Expected results (verify to 4 d.p.):
| Quantity | Value |
|---|---|
| $z_1$ | $[0.6500,\; 0.1000]$ |
| $h_1$ | $[0.5717,\; 0.0997]$ |
| $z_2$ | $[0.0972,\; 0.7442]$ |
| $h_2$ | $[0.0969,\; 0.6317]$ |
# Given values
W_xh = np.array([[0.5, 0.3], [-0.2, 0.4]])
W_hh = np.array([[0.1, -0.1], [0.2, 0.3]])
b_h = np.array([0.0, 0.1])
h_0 = np.array([0.0, 0.0])
x_1 = np.array([1.0, 0.5])
x_2 = np.array([-0.5, 1.0])
# TODO: Step 1 — compute h_1
# z_1 = W_xh @ x_1 + W_hh @ h_0 + b_h
# h_1 = np.tanh(z_1)
# TODO: Step 2 — compute h_2
# z_2 = W_xh @ x_2 + W_hh @ h_1 + b_h
# h_2 = np.tanh(z_2)
# Uncomment to verify:
# assert np.allclose(z_1, [0.65, 0.1], atol=1e-4), f'z_1 mismatch: {z_1}'
# assert np.allclose(h_1, [0.5717, 0.0997], atol=1e-4), f'h_1 mismatch: {h_1}'
# assert np.allclose(z_2, [0.0972, 0.7442], atol=1e-4), f'z_2 mismatch: {z_2}'
# assert np.allclose(h_2, [0.0969, 0.6317], atol=1e-4), f'h_2 mismatch: {h_2}'
# print(f'z_1 = {z_1}')
# print(f'h_1 = {h_1}')
# print(f'z_2 = {z_2}')
# print(f'h_2 = {h_2}')
# print('All hand-calculation checks passed. ✓')
Solution 1¶
# Solution 1 — executed verification of the hand calculation
z_1 = W_xh @ x_1 + W_hh @ h_0 + b_h
h_1 = np.tanh(z_1)
z_2 = W_xh @ x_2 + W_hh @ h_1 + b_h
h_2 = np.tanh(z_2)
assert np.allclose(z_1, [0.65, 0.1], atol=1e-4), f'z_1 mismatch: {z_1}'
assert np.allclose(h_1, [0.5717, 0.0997], atol=1e-4), f'h_1 mismatch: {h_1}'
assert np.allclose(z_2, [0.0972, 0.7442], atol=1e-4), f'z_2 mismatch: {z_2}'
assert np.allclose(h_2, [0.0969, 0.6317], atol=1e-4), f'h_2 mismatch: {h_2}'
print(f'z_1 = {np.round(z_1, 4)}')
print(f'h_1 = {np.round(h_1, 4)}')
print(f'z_2 = {np.round(z_2, 4)}')
print(f'h_2 = {np.round(h_2, 4)}')
print('All hand-calculation checks passed. \u2713')
z_1 = [0.65 0.1 ] h_1 = [0.5717 0.0997] z_2 = [0.0972 0.7442] h_2 = [0.0969 0.6317] All hand-calculation checks passed. ✓
Exercise 2 — Hand Derivation: BPTT Through a Scalar RNN¶
Make every chain-rule step of backpropagation through time visible by working with a scalar RNN (all weights and states are numbers, not matrices):
$$z_t = w_x\, x_t + w_h\, h_{t-1}, \qquad h_t = \tanh(z_t), \qquad h_0 = 0$$
Given: $w_x = 0.5$, $w_h = 0.8$, inputs $(x_1, x_2, x_3) = (1.0,\; -0.5,\; 0.25)$, target $y = 0.5$, loss $\mathcal{L} = \tfrac{1}{2}(h_3 - y)^2$.
Tasks (derive by hand, then verify numerically):
- Forward: compute $z_t$ and $h_t$ for $t = 1, 2, 3$ and the loss $\mathcal{L}$.
- Backward: define $\delta_t := \partial \mathcal{L} / \partial z_t$. Using the chain rule, show that $$\delta_3 = (h_3 - y)\,(1 - h_3^2), \qquad \delta_t = w_h\, \delta_{t+1}\, (1 - h_t^2) \quad (t < 3).$$ This is the scalar version of the Jacobian product from theory.md §3.3 — each backward step multiplies by $w_h \cdot \tanh'(z_t)$.
- Assemble the weight gradients (parameter sharing sums over time): $$\frac{\partial \mathcal{L}}{\partial w_h} = \sum_{t=1}^{3} \delta_t\, h_{t-1}, \qquad \frac{\partial \mathcal{L}}{\partial w_x} = \sum_{t=1}^{3} \delta_t\, x_t.$$
- Verify both gradients against central finite differences.
Expected results (verify to 4 d.p.):
| Quantity | Value |
|---|---|
| $(h_1, h_2, h_3)$ | $(0.4621,\; 0.1191,\; 0.2168)$ |
| $\mathcal{L}$ | $0.0401$ |
| $(\delta_3, \delta_2, \delta_1)$ | $(-0.2699,\; -0.2128,\; -0.1339)$ |
| $\partial \mathcal{L} / \partial w_h$ | $-0.1305$ |
| $\partial \mathcal{L} / \partial w_x$ | $-0.0950$ |
Solution 2¶
Forward (each step applies the recurrence):
$$z_1 = 0.5 \cdot 1.0 + 0.8 \cdot 0 = 0.5, \quad h_1 = \tanh(0.5) \approx 0.4621$$ $$z_2 = 0.5 \cdot (-0.5) + 0.8 \cdot 0.4621 \approx 0.1197, \quad h_2 \approx 0.1191$$ $$z_3 = 0.5 \cdot 0.25 + 0.8 \cdot 0.1191 \approx 0.2203, \quad h_3 \approx 0.2168$$ $$\mathcal{L} = \tfrac{1}{2}(0.2168 - 0.5)^2 \approx 0.0401$$
Backward. By the chain rule (derivative of $\tfrac{1}{2}u^2$, then of $\tanh$):
$$\delta_3 = \frac{\partial \mathcal{L}}{\partial h_3} \frac{\partial h_3}{\partial z_3} = (h_3 - y)(1 - h_3^2) \approx -0.2699$$
For $t < 3$, $z_t$ influences $\mathcal{L}$ only through $h_t \to z_{t+1}$ (chain rule through the recurrence, with $\partial z_{t+1}/\partial h_t = w_h$):
$$\delta_2 = \delta_3\, w_h\, (1 - h_2^2) \approx -0.2699 \cdot 0.8 \cdot 0.9858 \approx -0.2128$$ $$\delta_1 = \delta_2\, w_h\, (1 - h_1^2) \approx -0.2128 \cdot 0.8 \cdot 0.7864 \approx -0.1339$$
Weight gradients (sum over the three uses of each shared weight, $\partial z_t / \partial w_h = h_{t-1}$ and $\partial z_t / \partial w_x = x_t$):
$$\frac{\partial \mathcal{L}}{\partial w_h} = \delta_3 h_2 + \delta_2 h_1 + \delta_1 h_0 \approx -0.1305, \qquad \frac{\partial \mathcal{L}}{\partial w_x} = \delta_3 x_3 + \delta_2 x_2 + \delta_1 x_1 \approx -0.0950$$
Result: $\partial \mathcal{L}/\partial w_h \approx -0.1305$ and $\partial \mathcal{L}/\partial w_x \approx -0.0950$; note how each extra backward step multiplies $\delta$ by $w_h \tanh'(z_t) < 1$ — the scalar seed of the vanishing-gradient problem.
# Solution 2 — BPTT by hand, verified against central finite differences
w_x, w_h = 0.5, 0.8
xs = np.array([1.0, -0.5, 0.25])
y_target = 0.5
# Forward
hs = [0.0] # h_0
for x_t in xs:
hs.append(np.tanh(w_x * x_t + w_h * hs[-1]))
h1, h2, h3 = hs[1], hs[2], hs[3]
loss = 0.5 * (h3 - y_target) ** 2
# Backward: delta_t = dL/dz_t
delta3 = (h3 - y_target) * (1 - h3**2)
delta2 = delta3 * w_h * (1 - h2**2)
delta1 = delta2 * w_h * (1 - h1**2)
dL_dwh = delta3 * h2 + delta2 * h1 + delta1 * hs[0]
dL_dwx = delta3 * xs[2] + delta2 * xs[1] + delta1 * xs[0]
# Central finite differences on the full unrolled loss
def rnn_loss(wx, wh):
h = 0.0
for x_t in xs:
h = np.tanh(wx * x_t + wh * h)
return 0.5 * (h - y_target) ** 2
eps = 1e-6
fd_wh = (rnn_loss(w_x, w_h + eps) - rnn_loss(w_x, w_h - eps)) / (2 * eps)
fd_wx = (rnn_loss(w_x + eps, w_h) - rnn_loss(w_x - eps, w_h)) / (2 * eps)
# Deterministic checks against expected hand values
assert np.allclose([h1, h2, h3], [0.4621, 0.1191, 0.2168], atol=1e-4)
assert np.allclose(loss, 0.0401, atol=1e-4)
assert np.allclose([delta3, delta2, delta1], [-0.2699, -0.2128, -0.1339], atol=1e-4)
assert np.allclose(dL_dwh, -0.1305, atol=1e-4)
assert np.allclose(dL_dwx, -0.0950, atol=1e-4)
# BPTT must match finite differences tightly
assert np.allclose(dL_dwh, fd_wh, atol=1e-8), f'{dL_dwh} vs {fd_wh}'
assert np.allclose(dL_dwx, fd_wx, atol=1e-8), f'{dL_dwx} vs {fd_wx}'
print(f'h = ({h1:.4f}, {h2:.4f}, {h3:.4f}), L = {loss:.4f}')
print(f'delta = ({delta3:.4f}, {delta2:.4f}, {delta1:.4f})')
print(f'dL/dw_h = {dL_dwh:.6f} (finite diff: {fd_wh:.6f})')
print(f'dL/dw_x = {dL_dwx:.6f} (finite diff: {fd_wx:.6f})')
print('BPTT hand derivation matches finite differences. \u2713')
h = (0.4621, 0.1191, 0.2168), L = 0.0401 delta = (-0.2699, -0.2128, -0.1339) dL/dw_h = -0.130509 (finite diff: -0.130509) dL/dw_x = -0.094962 (finite diff: -0.094962) BPTT hand derivation matches finite differences. ✓
Exercise 3 — Coding: One LSTM Cell Forward Step¶
Implement a single LSTM forward step with the gate equations from
theory.md §5.3, using the same stacked-weight layout as
LSTMCell in first_principles.ipynb
(gate order f, i, g, o along the rows of $W$):
$$f_t = \sigma(z_f), \quad i_t = \sigma(z_i), \quad g_t = \tanh(z_g), \quad o_t = \sigma(z_o), \qquad \begin{bmatrix} z_f \\ z_i \\ z_g \\ z_o \end{bmatrix} = W\, [h_{t-1},\, x_t] + b$$
$$c_t = f_t \odot c_{t-1} + i_t \odot g_t, \qquad h_t = o_t \odot \tanh(c_t)$$
Specification: lstm_step(x, h_prev, c_prev, W, b) with
$W \in \mathbb{R}^{4m \times (m+d)}$, $b \in \mathbb{R}^{4m}$; returns (h, c).
Deterministic check — hand computation with $m = d = 1$, $h_{t-1} = 0$, $c_{t-1} = 0.5$, $x_t = 1$, and rows of $[W \mid b]$:
| Gate | $[w_h,\; w_x \mid b]$ | Pre-activation $z$ | Activation |
|---|---|---|---|
| forget | $[0.1,\; 0.4 \mid 1.0]$ | $1.4$ | $f = \sigma(1.4) \approx 0.8022$ |
| input | $[0.2,\; 0.3 \mid 0.0]$ | $0.3$ | $i = \sigma(0.3) \approx 0.5744$ |
| candidate | $[0.5,\; 0.8 \mid 0.0]$ | $0.8$ | $g = \tanh(0.8) \approx 0.6640$ |
| output | $[-0.3,\; 0.6 \mid 0.0]$ | $0.6$ | $o = \sigma(0.6) \approx 0.6457$ |
Expected results (verify to 4 d.p.):
$$c_t = 0.8022 \cdot 0.5 + 0.5744 \cdot 0.6640 \approx 0.7825, \qquad h_t = 0.6457 \cdot \tanh(0.7825) \approx 0.4224$$
Also verify the limiting behavior (theory §5.4): with $b_f$ large (forget gate saturated at 1) and $b_i$ very negative (input gate at 0), the cell state must pass through unchanged: $c_t \approx c_{t-1}$.
def sigmoid(z):
"""Numerically stable sigmoid."""
return np.where(z >= 0,
1 / (1 + np.exp(-z)),
np.exp(z) / (1 + np.exp(z)))
def lstm_step(x, h_prev, c_prev, W, b):
"""One LSTM forward step.
Args:
x: (d,) input at this time step
h_prev: (m,) previous hidden state
c_prev: (m,) previous cell state
W: (4m, m+d) stacked gate weights, row order f, i, g, o
b: (4m,) stacked gate biases
Returns:
(h, c): new hidden state and cell state, each (m,)
"""
# TODO: implement
# 1. concat = np.concatenate([h_prev, x])
# 2. gates = W @ concat + b, then slice into z_f, z_i, z_g, z_o
# 3. f, i = sigmoid(...); g = np.tanh(...); o = sigmoid(...)
# 4. c = f * c_prev + i * g; h = o * np.tanh(c)
pass
Solution 3¶
def lstm_step(x, h_prev, c_prev, W, b):
"""One LSTM forward step (gate order f, i, g, o)."""
m = h_prev.shape[0]
concat = np.concatenate([h_prev, x])
gates = W @ concat + b
f = sigmoid(gates[:m])
i = sigmoid(gates[m:2 * m])
g = np.tanh(gates[2 * m:3 * m])
o = sigmoid(gates[3 * m:])
c = f * c_prev + i * g
h = o * np.tanh(c)
return h, c
# --- Check 1: deterministic hand computation (m = d = 1) ---
W_hand = np.array([
[0.1, 0.4], # forget
[0.2, 0.3], # input
[0.5, 0.8], # candidate
[-0.3, 0.6], # output
])
b_hand = np.array([1.0, 0.0, 0.0, 0.0])
h_new, c_new = lstm_step(np.array([1.0]), np.array([0.0]), np.array([0.5]),
W_hand, b_hand)
assert np.allclose(c_new, 0.7825, atol=1e-4), f'c mismatch: {c_new}'
assert np.allclose(h_new, 0.4224, atol=1e-4), f'h mismatch: {h_new}'
print(f'Hand check: c_t = {c_new[0]:.4f}, h_t = {h_new[0]:.4f} \u2713')
# --- Check 2: limiting behavior — saturated forget gate, closed input gate ---
m_lim, d_lim = 3, 2
rng_ex3 = np.random.default_rng(7)
W_lim = rng_ex3.normal(size=(4 * m_lim, m_lim + d_lim)) * 0.1
b_lim = np.zeros(4 * m_lim)
b_lim[:m_lim] = 50.0 # forget gate -> sigmoid(~50) ~= 1
b_lim[m_lim:2 * m_lim] = -50.0 # input gate -> sigmoid(~-50) ~= 0
c_prev_lim = rng_ex3.normal(size=m_lim)
_, c_lim = lstm_step(rng_ex3.normal(size=d_lim), rng_ex3.normal(size=m_lim),
c_prev_lim, W_lim, b_lim)
assert np.allclose(c_lim, c_prev_lim, atol=1e-8), 'cell state should pass through'
print(f'Limiting check: f ~= 1, i ~= 0 => c_t == c_{{t-1}} '
f'(max diff {np.max(np.abs(c_lim - c_prev_lim)):.2e}) \u2713')
# --- Check 3: shapes and hidden-state range on a random configuration ---
h_r, c_r = lstm_step(rng_ex3.normal(size=d_lim), rng_ex3.normal(size=m_lim),
rng_ex3.normal(size=m_lim), W_lim, np.zeros(4 * m_lim))
assert h_r.shape == (m_lim,) and c_r.shape == (m_lim,)
assert np.all(np.abs(h_r) <= 1.0), 'h = o * tanh(c) must lie in [-1, 1]'
print('Shape and range checks passed. \u2713')
Hand check: c_t = 0.7825, h_t = 0.4224 ✓
Limiting check: f ~= 1, i ~= 0 => c_t == c_{t-1} (max diff 0.00e+00) ✓
Shape and range checks passed. ✓
Exercise 4 — Conceptual: Why Does LSTM's Cell State Act as a Gradient Highway?¶
Questions:
Vanilla RNN gradient path. In a vanilla RNN, the gradient from step $T$ to step $t$ passes through the product:
$$\prod_{s=t+1}^{T} \text{diag}(\tanh'(z_s))\, W_{hh}$$
Each factor includes both an element-wise nonlinearity ($\tanh'$) and the recurrent weight matrix $W_{hh}$. Explain why this causes vanishing gradients even when $\|W_{hh}\|$ is moderate. Relate your answer to the spectral-radius bound $(\gamma \cdot \lambda_{\max})^{T-t}$ from theory.md §4.2.
LSTM cell state gradient. In an LSTM, the gradient of $c_T$ w.r.t. $c_t$ is:
$$\frac{\partial c_T}{\partial c_t} = \prod_{s=t+1}^{T} f_s$$
where $f_s$ is the forget gate at step $s$. Explain why this is a "gradient highway" — why is this product better behaved than the vanilla RNN product?
When LSTM still fails. Give a concrete scenario where even an LSTM cannot learn a long-range dependency. What architectural modification addresses this?
Forget gate bias initialization. Why is $b_f$ often initialized to a positive value (e.g., 1.0)? What happens if it starts at 0?
# Numerical illustration: compare gradient products for RNN vs LSTM
T_steps = 50
m_dim = 8
rng_ex3 = np.random.default_rng(42)
# --- Vanilla RNN: product of diag(tanh'(z)) @ W_hh ---
W_hh_sim = rng_ex3.normal(size=(m_dim, m_dim)) * 0.3
rnn_grad_product = np.eye(m_dim)
rnn_norms = [np.linalg.norm(rnn_grad_product)]
for s in range(T_steps):
# Simulate tanh derivative at some random pre-activation
z_sim = rng_ex3.normal(size=m_dim)
tanh_deriv = 1 - np.tanh(z_sim) ** 2 # values in (0, 1]
jacobian = np.diag(tanh_deriv) @ W_hh_sim
rnn_grad_product = jacobian @ rnn_grad_product
rnn_norms.append(np.linalg.norm(rnn_grad_product))
# --- LSTM: product of forget gates ---
lstm_grad_product = np.ones(m_dim)
lstm_norms = [np.linalg.norm(lstm_grad_product)]
for s in range(T_steps):
# Simulate forget gate near 0.95 (learned to remember)
f_sim = 0.90 + 0.09 * rng_ex3.random(m_dim) # values in [0.90, 0.99]
lstm_grad_product = f_sim * lstm_grad_product
lstm_norms.append(np.linalg.norm(lstm_grad_product))
fig, ax = plt.subplots(figsize=(9, 5))
ax.semilogy(rnn_norms, 'o-', color='crimson', markersize=3, label='Vanilla RNN (Jacobian product)')
ax.semilogy(lstm_norms, 's-', color='steelblue', markersize=3, label='LSTM (forget gate product)')
ax.set_xlabel('Number of backward steps')
ax.set_ylabel('Gradient product norm (log scale)')
ax.set_title('Gradient Highway: LSTM forget gates vs RNN Jacobian products')
ax.legend()
ax.grid(True, alpha=0.3)
plt.tight_layout()
plt.show()
print(f'After {T_steps} steps:')
print(f' RNN gradient product norm: {rnn_norms[-1]:.2e}')
print(f' LSTM gradient product norm: {lstm_norms[-1]:.2e}')
print(f' Ratio (LSTM/RNN): {lstm_norms[-1] / (rnn_norms[-1] + 1e-30):.2e}')
print()
print('The LSTM cell state provides a much more stable gradient path.')
print('Key difference: the LSTM path involves only element-wise products')
print('of forget gates (scalars in [0,1]), while the RNN path involves')
print('full matrix multiplications that can compound shrinkage.')
rho = np.max(np.abs(np.linalg.eigvals(W_hh_sim)))
print()
print(f'Spectral radius of W_hh: rho = {rho:.3f}')
print('With gamma <= 1 from tanh saturation, gamma * rho < 1 predicts the')
print('exponential decay (gamma * rho)^(T-t) seen in the red curve above.')
After 50 steps: RNN gradient product norm: 1.51e-22 LSTM gradient product norm: 1.66e-01 Ratio (LSTM/RNN): 1.11e+21 The LSTM cell state provides a much more stable gradient path. Key difference: the LSTM path involves only element-wise products of forget gates (scalars in [0,1]), while the RNN path involves full matrix multiplications that can compound shrinkage. Spectral radius of W_hh: rho = 0.583 With gamma <= 1 from tanh saturation, gamma * rho < 1 predicts the exponential decay (gamma * rho)^(T-t) seen in the red curve above.
Solution 4¶
Vanilla RNN. Each backward step multiplies by $\text{diag}(\tanh'(z_s))\, W_{hh}$. Even if $\lambda_{\max}(W_{hh})$ is close to 1, the $\tanh'$ factors lie in $(0, 1]$ and are typically well below 1 (only $z = 0$ gives $\tanh' = 1$). The product norm behaves like $(\gamma \lambda_{\max})^{T-t}$ with effective $\gamma < 1$ (theory §4.2), so the gradient shrinks exponentially in the distance $T - t$ — moderate $\|W_{hh}\|$ does not prevent this because the shrinkage compounds multiplicatively at every step.
LSTM gradient highway. $\partial c_T / \partial c_t = \prod_s f_s$ is an element-wise product of learned gate values in $(0, 1)$ — no repeated multiplication by a weight matrix and no forced $\tanh'$ attenuation. The network can learn $f_s \approx 1$ on dimensions that must remember, making the product stay $\approx 1$ over long spans; a vanilla RNN has no comparable mechanism to hold its Jacobian at the identity.
When LSTM still fails. For very long ranges (e.g. a dependency spanning thousands of steps, as in document-level language modeling), even $f = 0.999$ gives $0.999^{5000} \approx 0.007$ — the highway still decays, and the single fixed-size cell state must compress everything in between. Attention/Transformers address this: any position attends directly to any other in $O(1)$ steps, removing the multiplicative path entirely (see ../16_transformer/theory.md).
Forget-gate bias. With $b_f = 0$ the gate starts at $\sigma(0) = 0.5$, so the untrained cell state is halved every step — after 20 steps only $0.5^{20} \approx 10^{-6}$ of early information (and gradient) survives, and the network may never receive a learning signal for long-range structure. Initializing $b_f = 1$ gives $f \approx \sigma(1) \approx 0.73$ (or higher with $b_f = 2$), keeping the highway open at the start of training (theory §5.6).