Skip to content

RL GridWorld Capstone

Tabular RL study on the library's GridWorldEnv (4x4 grid, goal at (3,3) worth +10, trap at (1,1) worth -5, -0.1 per step): train QLearningAgent across 5 seeds, sweep epsilon/alpha, and compare against an exact value-iteration solution computed from the same environment dynamics.

Setup

No dependencies beyond the repo root environment (ml_first_principles is installed editable; numpy/matplotlib/pytest come from the root requirements.txt):

# from the repo root
source .venv/bin/activate

How to run

# full experiment (~1.5s, deterministic) — writes reports/
python projects/rl_gridworld/src/rl_experiment.py

# tests (~1s)
python -m pytest projects/rl_gridworld/tests

# lint / format
ruff check projects/rl_gridworld
ruff format --check projects/rl_gridworld

Outputs land in reports/: report.md (full write-up with sweep table, greedy-policy arrow grids, findings), learning_curve.png (mean return across seeds with min-max band vs the value-iteration optimum), and value_heatmap.png (optimal \(V^\ast\) next to the learned \(\max_a Q\)).

Findings (details in reports/report.md)

  • Q-learning reaches the value-iteration optimum. Every trained configuration's greedy policy earns the optimal undiscounted return of 9.50 from the start state (shortest 6-step path, \(10 - 5 \times 0.1\)); a uniform-random policy earns -3.44.
  • Epsilon taxes the training return but not the final policy. Mean return over the last 50 episodes falls monotonically with exploration (9.14 at \(\epsilon=0.05\), 8.47 at \(0.1\), 7.18 at \(0.3\)) because random steps risk the trap; the best sweep config by tail return is \(\epsilon=0.05, \alpha=0.1\).
  • Alpha sets convergence speed. With deterministic dynamics a large step size is safe: \(\alpha=0.5\) makes the greedy policy optimal after ~13 episodes on average vs ~16 for \(\alpha=0.1\), with identical final quality.
  • Learned values match the exact solution along every visited optimal path; they undershoot only in rarely-visited cells near the trap (visible in the heatmap), which is expected for epsilon-greedy tabular Q-learning.

Structure

Path Contents
src/rl_experiment.py Training, sweep, value iteration, plots, report writing (main()).
tests/ 5 pytest tests: Bellman optimality, optimal rollout, improvement over random, policy agreement, determinism.
reports/ Generated report + figures (committed; regenerated byte-identically by the script).