Tried a recurrent architecture (HRM) for reasoning-retrieval, the bet held up.
The bet: BRIGHT is a retrieval benchmark where finding the right doc usually takes a few hops of reasoning, not just semantic overlap. Most embedders do a single forward pass. I wanted to see if a depth-recurrent architecture, one that loops over its own hidden state, would fit that better, so I built an embedder on HRM (Sapient's Hierarchical Reasoning Model). As far as I can tell it's the first time HRM's been used for retrieval.
The recurrence helped on the reasoning side, which was the whole bet. When I dialed the recurrence down at eval on pony (one of the BRIGHT domains), accuracy dropped with every loop I removed. Where it hit a wall was knowledge: the base was pretrained on a deliberately thin slice of text (Sapient built HRM-Text for pretraining efficiency, not breadth), so it's weak on knowledge-heavy domains. The part I find coolest: at 0.6B, the reasoning is coming from the architecture, not from scale.
Details:
- ~0.6B params, trained on one 3060 Ti (8GB).
- Recipe's deliberately boring: mean-pool + L2, bidirectional (LLM2Vec style), contrastive InfoNCE. Only the backbone is unusual. Same recipe as RakanEmbed4B.
Numbers (BRIGHT, mean nDCG@10, 12 domains):
- original: 18.1
- query rewriting: 34.3
- merged: 33.7
Weights are Apache-2.0 and the full BRIGHT eval harness is in the repo.
Open questions / discussion:
- Would a massively pretrained HRM push this further? The ceiling here looks like knowledge, not reasoning, so a broadly-pretrained base might lift it a lot. I don't have the compute to try that myself.
- Would other recurrent architectures show the same effect, or is something specific to HRM doing the work?
Model: https://huggingface.co/viventhraa96/HRM-Embed-0.6b
Code: https://github.com/okaybroda/hrm-embed
Full credits to Sapient Inc for open sourcing the code and the architecture for this work.