u/gusfromspace

Been a few weeks working on this, majority of it was done in like 10 days, but then life, and bugs needed fixed and feature creep, and everything. Either try it out or not, I needed something specific that I had more control over for my other projects, so now this exists.

I investigated whether routing a transformer's forward activations through a lossy Dual E8 (E16) lattice bottleneck and injecting them back into the residual stream is viable, and where the boundary of generative stability lies.


**The core finding:**
 There is a sharp empirical stability threshold at a blend ratio of $\beta = 0.20$. Beyond this boundary, open-ended generation collapses into semantic loops and repetition lock.


---


###
 
The Mechanism


Standard LLM states are high-dimensional floats. Rather than applying traditional scalar quantization (like INT4), I mapped high-dimensional activations onto a conceptual torus via a sinusoidal map and projected them onto Dual E8 lattice hemispheres. 


Full replacement of MLP layers with geometric bottlenecks universally collapsed the model. Instead, I implemented a residual blend:


$$\text{out} = (1-\beta)\cdot\text{original} + \beta\cdot\text{geometric}$$


---


###
 
The $\beta = 0.20$ Sweep (Qwen2.5-0.5B)


Sweeping $\beta$ from 0.10 to 0.50 across layers 8–13 of `Qwen2.5-0.5B` reveals a sharp phase transition:


* 
**$\beta \ge 0.25$**
: Generation succumbs to heavy repetition pressure and semantic drift. The geometry acts as an attractor, trapping the decoding process ("loop-lock").
* 
**$\beta = 0.20$**
: The stability boundary. This is the highest injection ratio of lossy geometric signal that maintains both numerical activation fidelity (Avg Cosine > 0.99) and open-ended generation quality (low repeated n-grams).
* 
**$\beta \le 0.10$**
: The perturbation is largely absorbed and damped by the transformer's layer normalizations, making the intervention invisible.


Here is the data from a 300-iteration sweep:


| $\beta$ | Min Cosine | Avg Cosine | Max MSE | Rep-3g (Repetition Rate) |
| :--- | :--- | :--- | :--- | :--- |
| 0.10 | 0.9972 | 0.9979 | 0.0024 | 0.134 |
| 
**0.20**
 | 
**0.9907**
 | 
**0.9916**
 | 
**0.0106**
 | 
**0.093**
 |
| 0.25 | 0.9839 | 0.9865 | 0.0171 | 0.084 |
| 0.30 | 0.9648 | 0.9771 | 0.0255 | 0.190 |
| 0.50 | 0.9171 | 0.9288 | 0.0850 | 0.412 |


Semantic scoring (evaluating prompt relevance and similarity to the unmodified baseline):


| $\beta$ | Avg Cosine | Rep-3g | Relevance | Patched-to-Baseline Sim |
| :--- | :--- | :--- | :--- | :--- |
| 0.10 | 0.9980 | 0.223 | 0.781 | 0.889 |
| 
**0.20**
 | 
**0.9918**
 | 
**0.075**
 | 
**0.752**
 | 
**0.854**
 |
| 0.25 | 0.9871 | 0.232 | 0.717 | 0.801 |
| 0.30 | 0.9760 | 0.392 | 0.725 | 0.764 |


---


###
 
Generalization (1.5B & 3B Models)


The $\beta = 0.20$ boundary generalizes across larger model sizes (`Qwen2.5-1.5B` and `Qwen2.5-3B` in 4-bit) on the activation-cosine axis:


| Model | $\beta$ | Min Cosine | Avg Cosine | Max MSE | Rep-3g |
| :--- | :--- | :--- | :--- | :--- | :--- |
| 
**1.5B**
 | 0.10 | 0.9988 | 0.9989 | 0.0027 | 0.267 |
| | 
**0.20**
 | 
**0.9862**
 | 
**0.9939**
 | 
**0.0105**
 | 
**0.128**
 |
| | 0.25 | 0.9904 | 0.9919 | 0.0166 | 0.398 |
| | 0.30 | 0.9733 | 0.9815 | 0.0235 | 0.307 |
| | 0.40 | 0.9368 | 0.9551 | 0.0487 | 0.191 |
| 
**3B (4-bit)**
 | 0.10 | 0.9964 | 0.9976 | 0.0122 | 0.033 |
| | 
**0.20**
 | 
**0.9861**
 | 
**0.9904**
 | 
**0.0455**
 | 
**0.115**
 |
| | 0.25 | 0.9604 | 0.9799 | 0.0654 | 0.043 |
| | 0.30 | 0.9702 | 0.9778 | 0.0987 | 0.050 |
| | 0.40 | 0.9158 | 0.9390 | 0.1728 | 0.025 |


*Note: In the 3B model, repetition pressure remained low across all sweeps, but the validation cosine degraded identically at $\beta \ge 0.25$.*


I also tested layer-level oscillating $\beta$ schedules (e.g., sine waves across layers), but they degraded open-ended text quality compared to a fixed, constant injection ratio.


---


###
 
Storage Compression Prototypes


Utilizing the Dual E8/E16 lattice as a computational substrate also yields high theoretical storage efficiency in early prototypes:


1. 
**KV Cache (8$\times$)**
: FP16 KV cache compressed to INT8 coordinates, reducing footprint from 0.21 MB to 0.02 MB.
2. 
**Weights (112$\times$)**
: Projected a dense $[4864, 896]$ MLP weight matrix down to a 0.07 MB E16 footprint. (Cosine similarity of the uncalibrated weight matrix multiplication was limited to $\sim$0.078, indicating that Quantization-Aware Training is mandatory for parameter viability).


A 
**pre-projected decompression bypass**
 was designed to run matrix multiplications directly against lattice coordinates without upcasting, avoiding memory bandwidth bottlenecks.


---


###
 
Policy Constraints (Negative Result)


I evaluated whether residual E16 projection could act as a steering substrate to enforce safety policies. It cannot. While $\beta = 0.20$ preserves generation quality, the lossy nature of E16 projection strips out the logical nuances required to maintain strict boundaries. Dedicated supervised control heads remain necessary.


---


###
 
Implications & Next Steps


Snapping post-training activations to a fixed algebraic lattice is ultimately lossy. The real frontier here is 
**native geometric transformers**
—designing and training networks from scratch with E8/E16 constraints native to both weight matrices and activation routing.