Built a 135M looped transformer with custom Muon+AdamW optimizer routing, per-sequence Poisson depth sampling, and truncated BPTT. Here's what the training code looks like.
Built a 135M dense looped LLM from scratch. Spent 2 weeks debugging Parcae's LTI stability mechanisms across 5 ablations. None of them beat the naive baseline at this scale. Trained for real anyway. SFT'd it. Shipped it. Here's the full honest story.
What I built
A 135M parameter looped transformer trained from scratch on FineWeb (4.6B tokens), inspired by the Parcae paper (arXiv:2604.12946 — "Scaling Laws For Stable Looped Language Models").
- 🤗 Base model: huggingface.co/harims95/LoopLM-135M-naive
- 🤗 SFT model: huggingface.co/harims95/LoopLM-135M-naive-sft
- 📂 Code: github.com/harims95/LoopLM
- 💰 Total cost: ~$51 (Modal H100s + free Lightning H200)
Architecture
Input → [Embedding] → [Prelude: 4 blocks] → e (injection)
→ [Loop block × T loops, T~Poisson(μ=6)] → [Coda: 2 blocks] → logits
- d_model 1024, GQA 16/8 heads, RoPE, QK-norm, SwiGLU FFN 2816
- Update rule:
h_{t+1} = block(h + e)(naive) or with LTI stability (Parcae) - Muon + AdamW optimizers, truncated BPTT (μ_bwd=3), bf16
- Trained on 2× H100 on Modal, ~3 hours wall clock
The Parcae investigation (the interesting part)
The paper claims LTI stability constraints on the recurrent state dramatically improve looped LM training. I tried to reproduce it. Here's what actually happened:
| Ablation | Description | Val loss |
|---|---|---|
| 1. Naive looped | h = block(h + e) |
3.84 |
| 2. + A matrix | LTI decay constraint | 3.84 (tied) |
| 3. + Input norm v1 | Wrong arch flow | Diverged |
| 4. + LTI before block | Fixed arch, B=identity | Worse |
| 5. + B→AdamW, init=0.447 | Matched official repo | Dramatically worse |
Every single "fix" — bringing my implementation closer to the official Parcae code — made things worse. After consulting:
- The paper's Appendix Q (optimizer routing)
- Official sandyresearch/parcae repo (injection.py)
- Two rounds of ChatGPT + Gemini debugging sessions
My conclusion: Parcae's stability improvements are a large-scale phenomenon. The paper's 1.3B model trains for 170k+ steps before stability mechanisms kick in. At 135M / 17.5k steps, naive looped is competitive enough that the extra complexity hurts more than it helps.
Comparison with sibling MoE
My brother built HobbyLM — a 500M MoE on the same infrastructure. For apples-to-apples comparison, I ran naive looped 135M on the same FineWeb data:
| Model | Architecture | Tokens | Val loss |
|---|---|---|---|
| LoopLM-135M (mine) | Dense looped | 4.6B | 3.95 |
| HobbyLM-130M MoE (bro) | Sparse MoE | 10B | 3.30 |
Dense looped loses to MoE at this scale/budget. Sparse MoE is more sample-efficient. Not surprising but now I have the data to confirm it.
SFT results (bonus)
Fine-tuned on Alpaca 52k using Lightning AI's free H200. Took 6 minutes (bf16 on H200 is insane).
Before SFT:
>
After SFT:
>
Improvement in format, not in facts. At 135M / 4.6B tokens, SFT teaches format, not knowledge. The model still hallucinates — that's a base model capacity problem, not a fine-tuning problem.
What I learned
On Parcae: Small-scale reproductions of large-scale papers are dangerous. The paper's key contribution (stability at 170k+ steps) is invisible at hobby budgets. Naive looped is a legitimate architecture for anyone training sub-1B models.
On MoE vs looped: At matched parameter count and token budget, MoE wins on sample efficiency. Looped models need more tokens to show their advantage, or need to be much bigger to amortize the loop cost.
On debugging: When 3 independent LLMs (me, ChatGPT 5.5, Gemini) all agree on a fix and it makes things worse — the paper's regime assumption is probably wrong, not your code.
On SFT: H200 on Lightning AI is free (2 hours/month) and runs 6 minutes of SFT for free. Use it. Colab Free disconnects at 3 hours. Don't use it for long jobs.
On honest publishing: val 3.95 is not impressive. The architecture exploration is. Shipping anyway with full documentation of what failed is more valuable than hiding failures.
Stack
- Training: Modal (H100s), Lightning AI (H200 for SFT)
- Framework: PyTorch, HuggingFace Transformers
- Optimizer: Muon (matrices) + AdamW (rest)
- Data: FineWeb via kjj0/fineweb10B-gpt2 shards
- Infra forked from: github.com/harishsg993010/HobbyLM (my brother's 500M MoE project)
Happy to answer questions about any part of this. The code is fully open, reproducible, and documented.
