MultiHashFormer: Hash-based Generative Language Models
We are excited to introduce MultiHashFormer, our new framework for vocabulary efficient language modelling.
Inspired by chaotic dynamic memory systems with distributed state spaces, we replace the traditional embedding matrix with a modular hashing interface.
👉 Each token is represented as a unique hash signature, a short sequence of discrete hash IDs, generated by multiple independent hash functions.
👉 A Hash Encoder compresses this ID signature into a single latent vector for processing by a Transformer decoder.
👉 A Hash Decoder generates the hash signature of the next token, which is then mapped back to text.
✅ Using 4 hash functions and 16,000 buckets per function, our model theoretically supports an upper bound of 16000^4 (approx. 65 quadrillion) unique signatures, i.e., vocabulary entries, with a constant memory footprint!
✅ MultiHashFormer consistently outperforms standard Transformer LMs across multiple benchmarks in 1B and 3B scales, pre-trained from scratch on 100B tokens (we know...we're compute poor, if you're interested in scaling further, please reach out).
✅ It can effectively handle multilingual vocabulary expansion with a constant parameter footprint without any architectural modifications or additional parameters!
Paper: https://arxiv.org/abs/2606.28057
HuggingFace: https://huggingface.co/papers/2606.28057