r/CUDA

Meta has published a fascinating deep dive into KernelEvolve, its agentic system for automatically generating, optimizing and validating production GPU kernels across NVIDIA, AMD and Meta's own MTIA accelerators:

https://engineering.fb.com/2026/04/02/developer-tools/kernelevolve-how-metas-ranking-engineer-agent-optimizes-ai-infrastructure/

I think the implications for AMD, and particularly for CUDA's long-term moat, are easy to underestimate.

A few highlights:

• KernelEvolve turns kernel optimization from a weeks-long expert engineering task into hours of automated search and evaluation.
• It generates code for NVIDIA GPUs, AMD GPUs, MTIA and CPUs from the same framework.
• It can target high-level DSLs such as Triton as well as low-level CUDA, HIP and MTIA C++.
• Every candidate is automatically compiled, tested for numerical correctness, benchmarked, profiled and fed back into the agent for another optimization round.
• Meta reports 100% correctness across 160 PyTorch ATen operators on three hardware platforms (480 configurations) and a 100% pass rate on all 250 KernelBench problems.
• This isn't just research: Meta says KernelEvolve is already optimizing production code serving trillions of inference requests per day.
• It delivered 60%+ inference throughput improvement on an NVIDIA production model and 25%+ training throughput improvement on MTIA.
• Perhaps most importantly, Meta says the system can optimize for hardware that wasn't even represented in the LLM's training data. Give the agent the architecture documentation, ISA, memory hierarchy and optimization rules, and it can learn to generate optimized kernels for the new accelerator.

That last point looks particularly important for AMD.

Historically, CUDA's moat wasn't simply "developers know CUDA." It was two decades of hand-written kernels, optimization knowledge, debugging, validation and tooling that made supporting another accelerator expensive and slow.

KernelEvolve attacks that engineering cost directly.

And there's an even more interesting data point in the paper: Meta now has more than 8,000 Triton kernels, surpassing its stagnant legacy CUDA kernel codebase.

So perhaps the biggest threat to CUDA isn't that AI agents will convert millions of lines of CUDA into ROCm.

It may be that increasingly new performance-critical code doesn't need to be written specifically for CUDA in the first place.

The emerging workflow could look more like:

Model/operator → high-level specification → agent → hardware-specific optimized kernel → automated correctness validation → production

with the agent generating the appropriate implementation for NVIDIA, AMD, MTIA or whatever accelerator comes next.

This also connects very nicely with AMD's recent strategy: ROCm becoming more mature, Meta's custom MI450-class AMD GPU, heterogeneous compute, and AMD's repeated argument that there is no "one-size-fits-all" solution in silicon.

CUDA obviously isn't going away. NVIDIA can use exactly the same agentic techniques and still has an enormous software ecosystem.

But if AI can compress weeks of hardware-specific expert work into hours, one of CUDA's greatest historical advantages — the sheer amount of engineering time already invested in it — may depreciate much faster than expected.

For AMD, the bar may therefore no longer be "make ROCm as mature as CUDA after 20 years."

It may simply become:

make AMD hardware sufficiently documented and accessible that autonomous optimization systems can do the rest.

Been working on this for the last few months. I've been working on PyTorch for the past few years and I always felt, many a times my work went into dump, because of some mistakes I made in the code. torch-preflight reads your PyTorch code and catches the bugs costing you GPU hours.

Things like losses.append(loss), which holds the autograd graph from every step until CUDA dies on you or no zero_grad() in the loop or gradient accumulation without dividing the loss or DDP with no DistributedSampler, so every rank trains on the same batches. I've been able to get 13 rules so far. Your code never gets imported or executed, so you need no GPU and no torch install.

There's another part to this that estimates VRAM. Point the tool at a training script and a GPU, and you learn whether the run fits before you pay for the instance. You also get the list of changes to make the run fit, with the GiB each one saves.

pip install torch-preflight

https://github.com/highwaterlabs/torch-preflight

https://pypi.org/project/torch-preflight/

Please try this out, and I would like to get your feedback! It's still a work in progeress.

Would like to know what breaks on your code. False positives kill a linter, and my only large test target so far has been the PyTorch source tree. Same for the memory numbers. Mine land within 4% of measured peaks, but from four models on one T4.

PS: open to contributions, and issues are already open on the repo. Soon I'm going to add a few "Good first issues" as well. Feel free to ping me if you have any questions!

I built and released an open-source Goldilocks/G64 STARK backend on AMD ROCm/HIP:

https://github.com/uulong950/qingming-stark-g64

The artifact exposes a complete proving boundary:

CLI prover → QSPG64 .qsp proof file → standalone verifier

The prover writes a real .qsp proof file. The standalone verifier reads that file and checks public input binding, statement digest, trace openings, quotient FRI, local AIR checks, and quotient relation checks.

The scale/latency boundary is the main point:

SCALE24: 2^24 rows, ~342 ms, verifier PASS
SCALE26: 2^26 rows, ~1.04 s, verifier PASS
SCALE27: 2^27 rows, ~2.04 s, fast_prelayout_xyz, verifier PASS

So this is not only a primitive benchmark. It is an open-source STARK backend producing standalone-verifiable proof files at million-row, sub-second scale on a consumer AMD GPU.

The build surface is small:

make -C rx7900xtx-24g

I am interested in what this latency/scale boundary makes possible:

local proving
proof-carrying APIs
low-cost prover markets
near-real-time verifiable computation
privacy-preserving business logic
hardware-neutral proving infrastructure

My current framing is:

SCALE24 = practical real-time region
SCALE27 = upper benchmark path

I would appreciate feedback on the artifact boundary and on what kinds of cryptographic systems could use open-source STARK proving at this scale.

We’ve been working on a runtime GPU optimization system at TETREVIS and have started publishing some of our NVIDIA Jetson benchmarking work publicly.

The approach operates at the execution/machine-code layer, and we’re now introducing dynamic runtime kernel fusion as part of the optimization pipeline.

Some of our current results:

Jetson Orin Nano — Qwen3.5 4B

Standard baseline: 10 → 21 tok/s (+110%)

CUDA Graphs baseline: 16 → 21 tok/s (+31.25%)

Jetson AGX Orin — Nemotron 3 Nano 4B

31.2 → 40.5 tok/s (~30%)

Jetson AGX Orin — Qwen3.5 4B

25.0 → 31.0 tok/s (+24%)

We’re currently expanding and stabilizing support across Jetson Orin Nano, Orin NX, and AGX Orin.

Rather than only posting performance claims, we’ve made the benchmarking repository available here:

https://github.com/mbuchel/sass2mlir-bench

The broader idea we’re exploring is whether more optimization can be moved to runtime — including machine-code optimization and dynamic kernel fusion — so the execution path can be adapted to the workload and GPU rather than relying entirely on what was determined ahead of execution.

There’s still quite a bit of work underway, particularly around consistency across the different Jetson configurations, but we wanted to start sharing the results and methodology publicly.

Technical feedback, criticism, and questions are welcome.

Built and open-sourced qingming-qwen3-tts, a device-native Qwen3-TTS inference backend for AMD RX 7900 XTX 24GB and NVIDIA RTX 4090 24GB.

For ROCm, the RX 7900 XTX backend uses native HIP kernels with BF16 storage and FP32 accumulation.

No PyTorch. No ONNX Runtime. No TensorRT. No CuBLAS. No rocBLAS.

Supports Qwen3-TTS 0.6B / 1.7B, streaming, Once and Resident execution.

The benchmark image uses a first audio packet of 8 codec frames = 640 ms of playable audio, so TTFA should not be directly compared with systems using 2-frame or 4-frame first packets.

GitHub: https://github.com/uulong950/qingming-qwen3-tts

I'm building a CUDA-native LLM runtime specifically for experimenting with GPU-level optimization on consumer GPUs, and I'd like some feedback.

The idea is basically a hackable Llama runtime where you can actually get into the CUDA kernels instead of fighting through a massive production inference stack.

The runtime is intended to let developers/researchers:

* Modify GEMM / Tensor Core kernels

* Experiment with FlashAttention and PagedAttention

* Tune KV-cache behavior

* Change tile sizes, memory layouts and thread configurations

* Experiment with kernel fusion and asynchronous execution

* Profile the resulting kernels

* Tune the runtime around the actual GPU they're running on

I'm doing this as my final-year engineering project, and I'm trying to determine whether this is actually useful to people who work with CUDA/LLMs/local inference.

2–3 minute survey:

https://forms.gle/KM4fUzVY1oC7g4TP8

If you've worked with CUDA, LLM inference, GPU optimization, llama.cpp, vLLM, TensorRT-LLM, FlashAttention, etc., I'd particularly appreciate your input.

Hey folks!

Did you ever wrapped model(x) in time.perf_counter() and gotten numbers that make no sense?

I realized it's a common enough trap and wrote a detailed write up here:

https://medium.com/traceopt/two-clocks-one-training-step-how-traceml-measures-pytorch-performance-357bc8e28dc7

TL;DR:

CUDA runs async. model(x) just enqueues kernels and returns, so a perf_counter() bracket around it measures how long Python took to queue the work, but not how long the GPU took to run it. The pending GPU time gets charged to whatever blocks next.

The tried the textbook fix, torch.cuda.synchronize() before each reading, which gives you accurate numbers but entirely about a different run.

Every sync becomes a stall, and it serializes exactly the CPU/GPU overlap you were trying to measure.

If one tires CUDA events (start.record() / end.record() / elapsed_time), it may fix both: the GPU stamps the markers as it passes, and you read them later with a non-blocking query() so nothing ever waits.

But i realized "CUDA events everywhere" is also wrong.

DataLoader next() is CPU work.

In a ML pipeline its time is high while the GPU's input wait is near zero, because the fetch overlaps the previous step.

Where I ended up: record both clocks for every phase, pick ONE clock per analysis window (and say which), report never-measured as null instead of 0.0, and only compare runs on a clock both measured.

How do you handle this in your own timing code: sync and eat the stall, or keep the two clocks separate?

cubisect is a small CUDA library for exact-match binary search. You give it a sorted float32 array and a batch of query values. It returns the index of each value, or -1 if the value is not there. Kernels compile at runtime through NVRTC. Torch tensors go in and out. CUDA tensors are passed by pointer, with no host copy.

I wrote this for fun and as an educational exercise for myself. The kernels are not necessarily optimal, and each one is probably not the best implementation of its method.

There are two ways to call the library, depending on whether the search is static (build once, query multiple times) or dynamic. The dynamic path searches the raw sorted array directly, with no setup. The static path reorders the array into a search-friendly layout once, then queries that layout.

Six kernels sit under those two paths, so they can be compared. The first four belong to the dynamic path:

• serial lower_bound. One thread per query, plain loop over the raw array in global memory. The baseline everything else is measured against.

• 2-way ILP in shared memory. Stages the whole array into shared memory, then each thread runs two queries at once so two independent load chains are in flight. Only usable while the array fits in shared memory.

• global memory with __ldg. Same two-queries-per-thread structure, but reads through the read-only cache instead of staging. This is the fallback when the array is too big for shared memory.

• XOR-swizzled shared memory. Same as the shared memory version, except shared addresses are permuted with the CuTe-like swizzle pattern, (row << 5) | ((col ^ row) & 31), to spread accesses across banks.

The last two belong to the static path:

• Eytzinger layout. The array reordered breadth-first, root at index 1 and the children of i at 2i and 2i+1. Stepping is just a shift, and the hot top of the tree sits in a few contiguous cache lines instead of being scattered across the array the way plain binary search midpoints are. It stores only the values; the index of a hit is recovered by accumulating subtree sizes on the way down, so there is no second index array.

• 32-ary warp-cooperative. One query per warp instead of one per thread. Each node holds 32 keys, one per lane, so a level is a single coalesced 128-byte load. __ballot_sync on key < query and __popc gives which of the 33 children to descend into. Depth is log32(n) instead of log2(n).

Which of the four dynamic kernels runs depends on the array size, and that choice comes from a lookup table. You generate the table by running python -m cubisect.autotune, which times all four kernels across a range of array sizes and records the fastest one at each size. If the top two are within 5% of each other, it marks that size a tie. The repo ships a table measured on my 5060, and you can regenerate it for your own GPU.

The static path has no autotuning. Building the layout costs time up front, so it only makes sense if you query the same array many times. It picks between its shared memory and global memory kernels based on whether the tree fits in shared memory.

Specs:

• GPU: RTX 5060 Laptop (sm_120)
• CPU: 24 threads, OpenMP for the baseline

Throughput in M queries/s, 2^20 queries per call, median of 7 runs. The static column is the Eytzinger kernel:

The two GPU paths cross over near 16k. Below that the raw array is faster, because it all fits in cache and the Eytzinger index arithmetic is just extra work. Above it the layout wins.

license is BSD-3, https://github.com/athrva98/cubisect

I built a 15-question AI Mastery Challenge on my platform to test who actually understands prompt engineering, multi-agent systems, and LLM behavior. 

 THE CONTEST: 

The person with the highest score on the leaderboard by next Sunday wins a $25 Cash Prize (or local equivalent) and a free permanent shoutout for their portfolio on our homepage!

How to enter:

1. Comment CHALLENGE below.

2. Below is the access link to the Quiz.

3. Take the quiz, register your username, and lock in your spot on the live leaderboard.

Quiz Link:

https://interconnectd.com/quiz/67/the-ultimate-ai-mastery-challenge-are-you-smarter-than-an-llm/

May the best prompt engineer win. Tag a friend who thinks they are an AI expert. 

#AI #PromptEngineering #GenerativeAI #LLM #NoCodeAI #IndieHackers #TechChallenge #ArtificialIntelligence #SoftwareEngineering #BuildInPublic