u/adityazero

That last dimension is not a number of elements. It's a number of bytes.

I went looking at how disaggregated inference hands a KV cache from prefill to decode — the thing AWS is doing with Trainium→Cerebras and AMD is doing with Helios→Cerebras.

The business slide version is simple: prefill produces a cache, ship it, decode consumes *it*. The problem is the word "it". Those 656 bytes decompose like this, per token:

→ 512 bytes: the compressed MLA latent, fp8
→ 16 bytes: four fp32 tile scales, written by specific warp lanes
→ 128 bytes: the RoPE component, bf16

A 576-element logical vector, stored in 656 bytes, across three regions of two dtypes, with quantization scales interleaved at a granularity determined by how a warp writes its lanes.

That isn't a tensor layout. It's a struct defined by a CUDA kernel, with load-bearing field offsets.

Zoom out and it's worse. Search vLLM for "def get_kv_cache_shape" and you find dozens of concrete implementations — diverging by backend, by attention variant, by model family, and by vendor.

As if that was not enough.

vLLM has a connector abstraction for shipping caches between instances. Read the signatures as going by the function name would get you in trouble:

```def save_kv_layer(self, layer_name: str, kv_layer: torch.Tensor, attn_metadata: "AttentionMetadata", ...)
```

This is what I call "strings based programming with duck-typing".

It abstracts the transport (UCX, RDMA, TCP, NVMe-oF, S3). It never abstracts the format so a Cerebras runtime can't implement that interface without reproducing vLLM's scheduler objects.

We built ELF so a linker from one vendor could consume objects from another. The KV cache has nothing like that and two vendor pairs (Cerebras + Trainium, Cerebras + Helios) just shipped products that need it.

https://hiraditya.github.io/posts/the-kv-cache-has-no-abi/

fp32 keeps about 7 significant digits.

When a running sum has reached 1.0, the smallest change fp32 can represent is one ULP ("unit in the last place") = 2⁻²³ ≈ 0.00000012.

Adding anything smaller than half of that — below 0.00000006 — rounds straight back to 1.0.

The addition happens and changes nothing.

That last dimension is not a number of elements. It's a number of bytes.

I went looking at how disaggregated inference hands a KV cache from prefill to decode — the thing AWS is doing with Trainium→Cerebras and AMD is doing with Helios→Cerebras.

The business slide version is simple: prefill produces a cache, ship it, decode consumes *it*. The problem is the word "it". Those 656 bytes decompose like this, per token:

→ 512 bytes: the compressed MLA latent, fp8
→ 16 bytes: four fp32 tile scales, written by specific warp lanes
→ 128 bytes: the RoPE component, bf16

A 576-element logical vector, stored in 656 bytes, across three regions of two dtypes, with quantization scales interleaved at a granularity determined by how a warp writes its lanes.

That isn't a tensor layout. It's a struct defined by a CUDA kernel, with load-bearing field offsets.

Zoom out and it's worse. Search vLLM for "def get_kv_cache_shape" and you find dozens of concrete implementations — diverging by backend, by attention variant, by model family, and by vendor.

As if that was not enough.

vLLM has a connector abstraction for shipping caches between instances. Read the signatures as going by the function name would get you in trouble:

```def save_kv_layer(self, layer_name: str, kv_layer: torch.Tensor, attn_metadata: "AttentionMetadata", ...)
```

This is what I call "strings based programming with duck-typing".

It abstracts the transport (UCX, RDMA, TCP, NVMe-oF, S3). It never abstracts the format so a Cerebras runtime can't implement that interface without reproducing vLLM's scheduler objects.

We built ELF so a linker from one vendor could consume objects from another. The KV cache has nothing like that and two vendor pairs (Cerebras + Trainium, Cerebras + Helios) just shipped products that need it.

https://hiraditya.github.io/posts/the-kv-cache-has-no-abi/

People keep talking about Cuda being a moat. And what makes it a moat is really the ptxas (the ptx assembler that converts ptx to sass binary). With current technologies it seems possible to make a custom ptx compiler but I wonder if this effort is worth someone's time.

There is one thing about performance that I feel can be unlocked with such a tool but I am yet to find a good test case for that.

I built vLLM from source on Ubuntu 26.04 recently.

According to the official docs, a simple `pip install -e .` should have been enough. In reality, I (+Claude) hit a chain of version-skew, driver, and toolchain issues. From silent driver conflicts to CUDA toolkit mismatches that only fail at runtime, this build taught me things that the documentation left out.

If you are tackling a fresh GPU build on a recent OS, check out this field guide. It covers:
• Hardware detection: nvidia-smi vs lspci commands
• Fixing elusive runtime errors and device node issues
• Resolving ptxas failures by matching toolkit versions
• Handling flash-attention Python version checks

Don't let cryptic error messages waste your day.
https://hiraditya.github.io/posts/building-vllm-from-source/

Pro tip: If you don't set `TORCH_CUDA_ARCH_LIST` to match your specific GPU, you'll either wait 5–10x longer for the build (compiling every architecture) or hit cryptic and irrelevant CUDA errors.

For a long time, the biggest pain point in JAX was that it would happily insert an expensive all-gather because your sharding was technically valid but logically inefficient. Or worse, you'd miss an all-reduce and get wrong numbers... silently.

I sat down to write about that gap. By the time I finished, it had mostly closed, and finding out how thoroughly it stopped being true was the most interesting part of writing this.

The post is a tour of the three ways JAX handles placement:
- Auto (GSPMD infers, and the type stays quiet),
- Explicit (placement moves into the type, and mismatches fail at trace time),
- Manual (shard_map, where you write the collectives yourself)

I walk through each with real jaxprs, HLO, and MLIR from an 8-device run to show what actually happens under the hood.

The compiler is no longer just a black box; it’s an active partner in correctness.

https://hiraditya.github.io/posts/how-jax-shards-a-computation/

Nvidia doesn't seem to publish a specification for the binary format of a CUDA kernel, section layout, or the constant bank parameter conventions. So I dug into it.

A `.cubin` is an ELF64 executable with a flat stream of undocumented "EIATTR" attributes that encode everything the driver needs to launch a kernel: register count, parameter layout, EXIT instruction offsets, and constant bank geometry.

`.nv.info`  uses an undocumented TLV encoding to make kernels self-describing register counts, parameter offsets, EXIT locations are all serialized into a flat byte stream the driver parses at load time.

And the constant bank parameter base is not an architectural constant. It has changed silently across toolkit versions from Ampere to Hopper to Blackwell. The fact that the code, the  `.nv.info`  metadata , and the  `.nv.constant0`  section size all encode the parameter base offset independently surprised me.

The post discovers the section layout, the EIATTR encoding, symbol table conventions, and the note sections the driver validates before loading on a B200 silicon.

https://hiraditya.github.io/posts/anatomy-of-a-cuda-binary/

Nvidia doesn't seem to publish a specification for the binary format of a CUDA kernel, section layout, or the constant bank parameter conventions. So I dug into it.

A `.cubin` is an ELF64 executable with a flat stream of undocumented "EIATTR" attributes that encode everything the driver needs to launch a kernel: register count, parameter layout, EXIT instruction offsets, and constant bank geometry.

`.nv.info`  uses an undocumented TLV encoding to make kernels self-describing register counts, parameter offsets, EXIT locations are all serialized into a flat byte stream the driver parses at load time.

And the constant bank parameter base is not an architectural constant. It has changed silently across toolkit versions from Ampere to Hopper to Blackwell. The fact that the code, the  `.nv.info`  metadata , and the  `.nv.constant0`  section size all encode the parameter base offset independently surprised me.

The post discovers the section layout, the EIATTR encoding, symbol table conventions, and the note sections the driver validates before loading on a B200 silicon.

https://hiraditya.github.io/posts/anatomy-of-a-cuda-binary/