Trained a ~20K LM (probably smallest) that can still write stories
I’ve been pushing TinyStories-style models downward in size, and this is the smallest one so far:
MacroStories — 19,969 parameters, 81 KB FP32
https://huggingface.co/raincandy-u/MacroStories
For scale:
→ ~50× smaller than the 1M TinyStories model
→ ~3,000× smaller than AlexNet
→ 32-dim hidden state
→ 378-token vocabulary
→ one decoder block, recurrently applied 4 times with shared weights
It’s obviously not a general-purpose LM, but within its constrained story distribution it can maintain a 100–300 word narrative with a goal, problem, relevant actions, and resolution.
It also runs extremely fast on CPU and needs no GPU.
I’m mostly interested in how far the lower bound for coherent narrative generation can be pushed.
Would be curious to see how people manage to break it.☺️