Gradient Descent
Phase: 1 | Status: ✅ Complete | Prerequisites: 01 Linear Regression, Calculus & Optimization
Overview
The foundational iterative optimization algorithm for machine learning. Covers full-batch GD, SGD, mini-batch SGD, momentum (Polyak heavy-ball, Nesterov), adaptive methods (AdaGrad, RMSProp, Adam), learning rate schedules, convergence theory, and failure modes.
Contents
| # | File | Type | Description |
|---|---|---|---|
| 1 | theory.md |
Theory | GD update rule, convergence rates (convex \(O(1/T)\), strongly convex linear rate), smoothness condition \(\alpha < 2/M\), momentum, adaptive methods, schedules |
| 2 | first_principles.ipynb |
Computation | From-scratch GD, momentum, Adam; learning rate effects; convergence race; library comparison |
| 3 | exercises.ipynb |
Practice | Hand-calc GD steps, implement momentum GD, conceptual questions on Adam vs SGD |
Connections
- Prereqs: 01 Linear Regression, Calculus & Optimization
- Synthesis: Optimization Methods Compared
- Next: 03 Regularization, 04 Logistic Regression
- Used by: Every iterative ML algorithm — logistic regression, SVM, neural networks, ensemble boosting