
Using Poincaré hyperbolic geometry to solve a volume scaling problem in neural network interpretability
Wanted to share an interesting application of hyperbolic geometry to machine learning interpretability.
The setup: Sparse Autoencoders decompose neural network activations into interpretable features. These features are dictionary atoms embedded in R^(d.) The problem is that the concepts networks learn form branching hierarchies (trees), and trees with branching factor b have O(b^(r)) nodes at depth r. But the volume of a Euclidean ball grows as O(r^(d)) -- polynomially.
This mismatch means that at large dictionary sizes (16K+), there isn't enough Euclidean volume for features to spread out. They collide at the boundary and "die" (stop activating).
The fix: embed dictionary weights in the Poincaré ball model of hyperbolic space, where the volume element grows as sinh^(d-1)(r) ~ O(e^(r).) This matches the exponential branching of concept hierarchies.
The interesting constraint: the forward pass of the autoencoder must stay Euclidean (for compatibility with the host neural network's normalization layers). So the hyperbolic embedding is applied only as a training-time weight regularizer via an entailment cone loss on the Poincaré-projected dictionary atoms.
Empirically, this reduces dead features from 3.8% to 0.2% and improves reconstruction by 9.8% on a 2B-parameter language model.
Paper: https://vishalvermalabs.com/papers/empirical-validation-hypersae-poincare-geometry/ Code: https://github.com/vishal-dehurdle/hypersae