Sentinel - A WIP C++/CUDA LLM Training Framework Made For Consumer Hardware
Hello!
I'm not to sure what to say since this is my first time writing about this framework.
I've been making (with the help of AI agents as senior devs, please dont judge i haven't "vibe coded" anything) a framework for a little while now which is made for full training / full fine tuning.
It is still under development so please do not expect it to be perfect or well tested on different gpu's \*yet\*
Specs:
GPU: 5070 ti (16GiB VRAM)
CPU: Ryzen 7 7800x3d
RAM: 32GB DDR5
| Model | tok/s | Peak VRAM | Host RAM |
| :--- | :---: | | :---: | ---: |
| \~100M | 20,013 (20k) | \~3.35GiB | \~1.45GiB |
| \~500M | 6.162 (6k) | \~4.71GiB | \~4.62GiB |
| \~1B | 3.085 (3k) | \~6.81GiB | \~10.64GiB |
| \~1.5B | 1.887 (1.8k) | \~8.5GiB | \~18.5GiB |
| \~4B | 1.116 (1.1k) | \~12.3GiB | \~15.9GiB |
The interesting part isnt even necessarily the raw tok/s. The goal is to make full training of models that normally dont comfortably fit into consumer GPU memory practical through things like GPU/host offloading and memory efficient training paths.
Sentinel is written primarily in C++/CUDA, with python bindings, and im currently working on improving the optimizer and kernel performance further.
Its very much a work in progress as said already, but i thought id finally share it and see what people think.
