u/zero_planck

WISP v1.1 — 121 tests, three new features:

→ KDA attention kernel: Kimi K3's linear attention

implemented in CUDA + PyTorch fallback.

State is constant-size regardless of context.

On-device verified against double-precision reference.

→ Learning cache: WISP now records expert usage

across sessions. Next startup pre-warms hot experts.

Verified on real Mixtral: 768 observations →

238 tracked → 107 pre-warmed on session 2.

Gets faster the more you use it.

→ OpenAI API: wisp serve --port 8080

Verified live over real HTTP.

Works with Cursor, Continue.dev, Open WebUI.

Streaming is genuinely incremental.

Also fixed: expert routing was invisible to Python

since day one. Prefetch predictor now sees real data.

121 tests. CI green. MIT.

github.com/zeroextub-collab/wisp

WISP v1.1 — 121 tests, three new features:

→ KDA attention kernel: Kimi K3's linear attention

implemented in CUDA + PyTorch fallback.

State is constant-size regardless of context.

On-device verified against double-precision reference.

→ Learning cache: WISP now records expert usage

across sessions. Next startup pre-warms hot experts.

Verified on real Mixtral: 768 observations →

238 tracked → 107 pre-warmed on session 2.

Gets faster the more you use it.

→ OpenAI API: wisp serve --port 8080

Verified live over real HTTP.

Works with Cursor, Continue.dev, Open WebUI.

Streaming is genuinely incremental.

Also fixed: expert routing was invisible to Python

since day one. Prefetch predictor now sees real data.

121 tests. CI green. MIT.

github.com/zeroextub-collab/wisp

I built WISP — a CUDA engine for streaming 744B+ parameter MoE models on consumer hardware

Last week I found Colibrì by JustVugg, a ~2,400-line pure-C engine exploring a crazy idea:

What if you don't load the entire model into RAM?

MoE models only activate a fraction of their parameters for each token. So instead of trying to fit hundreds of billions of parameters in memory, you can stream the experts the model actually needs.

That idea sent me down a rabbit hole.

I built WISP — Stream What Shouldn't Run.

The architecture is basically:

Token
  ↓
Model router selects experts
  ↓
VRAM cache → hit? use it
  ↓
RAM cache → hit? transfer it
  ↓
NVMe → stream cold expert
  ↓
LRU promotes frequently used experts

The goal is to turn VRAM + RAM + NVMe into one memory hierarchy for MoE inference.

WISP adds a few things on top of the original streaming concept:

CUDA acceleration for attention/FFN compute, a C hot path for expert loading and caching, and Python for orchestration.

Absorbed MLA for architectures like DeepSeek, keeping the compressed latent representation instead of storing fully expanded K/V tensors.

Double-buffered async streaming, so CPU/I/O can prepare expert data while the GPU is working instead of making the GPU sit around waiting for storage.

Speculative decoding, using a smaller same-family model to draft tokens while the target model verifies them.

Hardware auto-configuration, which profiles VRAM, RAM, storage throughput, etc. and calculates the cache split automatically.

I tested the current engine with Mixtral-8x7B on:

Ryzen 7 9800X3D
RTX 5070 12GB
32GB DDR5-6000
PCIe 4.0 NVMe (~4.34 GB/s)

Current measured result:

0.75 tok/s cold

After only 80 tokens, the expert cache reached a 68.8% hit rate.

Mixtral does 64 expert activations/token (2 experts × 32 layers), and all 256 experts in my tested representation occupy ~14.3GB, so once they're warm in RAM the engine can stop doing cold SSD expert reads.

The biggest thing I learned building this:

The bottleneck isn't necessarily CUDA. It's bytes moved per token.

I spent time thinking GPU kernels would be the main optimization target.

Then you realize shaving milliseconds off a matmul doesn't matter much when your runtime is waiting for a giant expert to come off NVMe.

Cache locality, expert size, storage bandwidth and I/O overlap become insanely important.

And that's why I'm particularly interested in testing this architecture on much larger MoE models with smaller individual experts.

The project currently targets:

GLM-5.2       744B
DeepSeek-V3   671B
DeepSeek-R1   671B
Mixtral-8x7B   47B
Mixtral-8x22B 141B

Future targets:
Kimi K3
Qwen3.8

And yes, huge credit to JustVugg / Colibrì.

Colibrì demonstrated the core streaming concept. WISP is my attempt to generalize it into a multi-model runtime with CUDA, hierarchical caching, MLA support, async streaming and speculation.

Colibrì:
github.com/JustVugg/colibri

WISP:
github.com/zeroextub-collab/wisp

MIT licensed. 73 tests passing.

Still experimental, and I'm deliberately separating measured numbers from projected ones.

I'm especially interested in feedback from people working on CUDA, inference runtimes, MoE routing, quantization, or storage/I/O optimization.

What would you optimize first: expert prediction/prefetching, cache policy, quantization, or the I/O pipeline?

I built WISP — a CUDA engine for streaming 744B+ parameter MoE models on consumer hardware

Last week I found Colibrì by JustVugg, a ~2,400-line pure-C engine exploring a crazy idea:

What if you don't load the entire model into RAM?

MoE models only activate a fraction of their parameters for each token. So instead of trying to fit hundreds of billions of parameters in memory, you can stream the experts the model actually needs.

That idea sent me down a rabbit hole.

I built WISP — Stream What Shouldn't Run.

The architecture is basically:

Token
  ↓
Model router selects experts
  ↓
VRAM cache → hit? use it
  ↓
RAM cache → hit? transfer it
  ↓
NVMe → stream cold expert
  ↓
LRU promotes frequently used experts

The goal is to turn VRAM + RAM + NVMe into one memory hierarchy for MoE inference.

WISP adds a few things on top of the original streaming concept:

CUDA acceleration for attention/FFN compute, a C hot path for expert loading and caching, and Python for orchestration.

Absorbed MLA for architectures like DeepSeek, keeping the compressed latent representation instead of storing fully expanded K/V tensors.

Double-buffered async streaming, so CPU/I/O can prepare expert data while the GPU is working instead of making the GPU sit around waiting for storage.

Speculative decoding, using a smaller same-family model to draft tokens while the target model verifies them.

Hardware auto-configuration, which profiles VRAM, RAM, storage throughput, etc. and calculates the cache split automatically.

I tested the current engine with Mixtral-8x7B on:

Ryzen 7 9800X3D
RTX 5070 12GB
32GB DDR5-6000
PCIe 4.0 NVMe (~4.34 GB/s)

Current measured result:

0.75 tok/s cold

After only 80 tokens, the expert cache reached a 68.8% hit rate.

Mixtral does 64 expert activations/token (2 experts × 32 layers), and all 256 experts in my tested representation occupy ~14.3GB, so once they're warm in RAM the engine can stop doing cold SSD expert reads.

The biggest thing I learned building this:

The bottleneck isn't necessarily CUDA. It's bytes moved per token.

I spent time thinking GPU kernels would be the main optimization target.

Then you realize shaving milliseconds off a matmul doesn't matter much when your runtime is waiting for a giant expert to come off NVMe.

Cache locality, expert size, storage bandwidth and I/O overlap become insanely important.

And that's why I'm particularly interested in testing this architecture on much larger MoE models with smaller individual experts.

The project currently targets:

GLM-5.2       744B
DeepSeek-V3   671B
DeepSeek-R1   671B
Mixtral-8x7B   47B
Mixtral-8x22B 141B

Future targets:
Kimi K3
Qwen3.8

And yes, huge credit to JustVugg / Colibrì.

Colibrì demonstrated the core streaming concept. WISP is my attempt to generalize it into a multi-model runtime with CUDA, hierarchical caching, MLA support, async streaming and speculation.

Colibrì:
github.com/JustVugg/colibri

WISP:
github.com/zeroextub-collab/wisp

MIT licensed. 73 tests passing.

Still experimental, and I'm deliberately separating measured numbers from projected ones.

I'm especially interested in feedback from people working on CUDA, inference runtimes, MoE routing, quantization, or storage/I/O optimization.

What would you optimize first: expert prediction/prefetching, cache policy, quantization, or the I/O pipeline?

Body: Kimi K3 dropped Thursday. Qwen3.8 announced yesterday. Both 2T+ MoE models. Both need a streaming engine. WISP is that engine. What it does: → 3-tier streaming: VRAM → RAM → NVMe SSD → Self-organizing LRU cache (no config needed) → Absorbed MLA attention (~70KB/token KV cache) → Same-family speculative decoding (2.2-2.8x throughput) → Auto-configures any hardware automatically → Display auto-detection (prevents GPU black screen) → RAM watermark (never OOMs) Verified: Mixtral-8x7B generating coherent code on RTX 5070 12GB, 0.75 tok/s cold, 68.8% cache hit rate after 80 tokens. GLM-5.2 is where it truly sings — 17.5MB experts vs 99MB for Mixtral = 5.7x faster. 73 tests. MIT license. C + CUDA + Python. Inspired by Colibrì (JustVugg). github.com/zeroextub-collab/wisp