![[Research] BCMT: Blockwise Causal Memory Transformer](https://external-preview.redd.it/0krlBW8mcrhQyWjtiDkjKNLoDF82siym_wo2IGsFukw.png?width=1080&crop=smart&auto=webp&s=b15a60d0b7c995c84e87d15003c66c979a46949f)
[Research] BCMT: Blockwise Causal Memory Transformer
Hi everyone,
I'd like to share a research project I've been working on for the past few months.
BCMT (Blockwise Causal Memory Transformer) explores an alternative approach to long-context language modeling.
Instead of propagating long-range information through dense global self-attention, BCMT keeps dense causal self-attention within local blocks and propagates contextual information through a compact exponential causal memory built from adaptive block summaries.
The goal is to investigate whether long-range dependencies can be modeled efficiently while reducing the computational and memory costs associated with global attention.
The paper presents:
- the complete BCMT architecture,
- the mathematical formulation,
- an open-source PyTorch implementation,
- an initial experimental evaluation on WikiText language modeling.
The initial experiments show validation perplexities close to a dense Transformer baseline while achieving higher training throughput and lower GPU memory usage.
I'd greatly appreciate technical feedback on the architecture, the experimental methodology, or potential limitations. Any technical feedback, suggestions, or criticism would be greatly appreciated.
Code: https://github.com/rachidlabs/BCMT
Paper (DOI): https://doi.org/10.20944/preprints202607.0333.v1