Built GPT-2 on Custom Deep Learning Framework I built from scratch in C++
since jan 2026 i've been building Forge, a deep learning framework written entirely from scratch in C++ - no PyTorch, no TensorFlow underneath.
Eigen handles most of the math backend. btw i wrote some custom AVX2 SIMD kernels (element-wise ops) too, and OpenBLAS-backed GEMM for the heavy matrix ops.
what's implemented so far:--
- A custom tensor engine with its own autodiff engine and memory allocator
- Dense/Linear layers, Optimizers (Adam, AdamW, SGD and SGD with momentum), Self Attention, LayerNorm, Activation Functions (sigmoid, softmax, tanh, GELU[tanh approximation], RELU, leakyRELU), loss functions (Cross Entropy Loss [log softmax fused], Binray Cross Entropy (Sigmoid fused), and Mean Squared Error) and Embeddings.
- A from-scratch BPE tokenizer (GPT-2-style pre-tokenization + merges)
- A reflection-based (reflect-cpp) parameter system - models declare their structure, Forge auto-discovers trainable parameters, no manual registration
- a safetensors-format save/load pipeline
the part I'm actually proud of- I loaded real pretrained GPT-2 small weights into a GPT-2 architecture built entirely on Forge, and under greedy decoding, its output matches HuggingFace's transformers library token-for-token. not similar, but exact. every layer (embeddings, attention, LayerNorm, the final projection) has to be numerically correct for that to hold, since a single wrong transpose or masking bug would have diverged the output within a few tokens.
it's still CPU-only for now (currently limited to float32 and int32 - working through some dtype/SIMD coverage gaps), and slower than i'd like (the only main culprits are the CE loss fn implementation and its gardient function and softmax, which i am on to optimize, it has no KV-cache yet) - a CUDA backend and those perf fixes are next on the list.
Repo: https://github.com/muchlakshay/Forge
Windows/Linux release builds: https://github.com/muchlakshay/Forge/releases/tag/0.1
YT demo link - https://www.youtube.com/watch?v=EO1aYBF5jwU
would love feedback, especially from anyone who's built something similar and much better than me.
thats all. im a 17yo deeply passionate about Deep Learning and system level programming.