I built using claude a 35-stage course where you reimplement PyTorch from scratch — no autograd libraries allowed
I kept noticing that I could use PyTorch fine but couldn't actually explain what .backward() does under the hood. So I built a project-based curriculum to fix that for myself, and cleaned it up enough to share.
The idea: you rebuild a deep learning framework from zero, one concept at a time. The only libraries you're allowed are NumPy (for forward array math — never to compute a gradient for you), Matplotlib, and pytest. No torch, no autograd, no micrograd. The rule is: you don't get to import a concept until you've built it by hand in an earlier stage. You are the autodiff library.
How it's structured — 35 stages, each a folder with exactly 3 files:
README.md— the intuition, the key gradient equations, a video or two to watch, and one unambiguous exercisecode.py— a skeleton: full interfaces, docstrings, and TODOs, but no working bodiestest.py— pytest tests, including numerical gradient checks (central differences) so you know your backward pass is correct, not just plausible
You fill in code.py until pytest goes green, then move to the next stage. Crucially, each stage imports and extends the code you wrote in earlier stages — so the framework genuinely grows under your hands instead of being 35 disconnected toy scripts.
The arc:
scalar backprop → a reverse-mode autodiff engine → an N-dimensional Tensor → layers, losses, optimizers (SGD/momentum/Adam) → a real training loop → BatchNorm/Dropout → CNNs (Conv2d via im2col, with the backward derived by hand) → attention → a full Transformer → a Vision Transformer → packaging it all into a small PyTorch-like framework → capstone projects.
By the end you've written every gradient and every chain-rule accumulation yourself.
It's free and open source. Feedback very welcome — especially if you work through a stage and find something unclear or a test that feels off.
👉 https://github.com/roiamiel1/Build-Deep-Learning-From-Scratch