u/checkmydoor

We have published the initial technical findings from our SASS2MLIR work.

The project explores GPU optimization at a layer below conventional framework- and compiler-level tuning, including analysis and transformation of the machine code ultimately executed by the GPU.

Across our testing so far, we have observed ~20% to 100%+ performance improvements, depending on the architecture, kernel, workload, and execution conditions.

Testing has included:

NVIDIA architectures spanning Ampere through Blackwell

Jetson Orin Nano, Orin NX, and AGX Orin

Individual instruction and microbenchmark testing

Kernel-level benchmarking

Model and workload-level testing

Comparisons against conventional execution paths, including CUDA Graphs in applicable tests

One of the areas we are particularly interested in is the optimization opportunity that exists after traditional compilation has already taken place.

Our broader work looks at analyzing the final GPU machine code, identifying architectural and execution inefficiencies, and dynamically modifying the execution path while maintaining numerical correctness.

This includes areas such as instruction scheduling, memory behavior, register utilization, execution dependencies, architecture-specific instruction behavior, and increasingly runtime kernel optimization and dynamic kernel fusion.

The interesting result for us is that the performance opportunity is not limited to a single GPU generation or workload type. We are seeing measurable opportunities across both datacenter-class GPUs and constrained edge platforms such as Jetson, although the magnitude of the improvement varies considerably with workload characteristics and hardware limits.

These are still initial findings, and we are continuing to expand the benchmark coverage and validate the methodology across additional models, architectures, and workloads.

For anyone interested in the deeper engineering details, we have published a technical explanation of the discoveries, methodology, and underlying work here:

https://mbuchel.github.io/projects/sass2mlir/

Technical feedback, criticism, and discussion are very welcome.

(And to save time I did have AI rewrite this more professionally:) )

We’ve been working on a runtime GPU optimization system at TETREVIS and have started publishing some of our NVIDIA Jetson benchmarking work publicly.

The approach operates at the execution/machine-code layer, and we’re now introducing dynamic runtime kernel fusion as part of the optimization pipeline.

Some of our current results:

Jetson Orin Nano — Qwen3.5 4B

Standard baseline: 10 → 21 tok/s (+110%)

CUDA Graphs baseline: 16 → 21 tok/s (+31.25%)

Jetson AGX Orin — Nemotron 3 Nano 4B

31.2 → 40.5 tok/s (~30%)

Jetson AGX Orin — Qwen3.5 4B

25.0 → 31.0 tok/s (+24%)

We’re currently expanding and stabilizing support across Jetson Orin Nano, Orin NX, and AGX Orin.

Rather than only posting performance claims, we’ve made the benchmarking repository available here:

https://github.com/mbuchel/sass2mlir-bench

The broader idea we’re exploring is whether more optimization can be moved to runtime — including machine-code optimization and dynamic kernel fusion — so the execution path can be adapted to the workload and GPU rather than relying entirely on what was determined ahead of execution.

There’s still quite a bit of work underway, particularly around consistency across the different Jetson configurations, but we wanted to start sharing the results and methodology publicly.

Technical feedback, criticism, and questions are welcome.

We’ve published our initial SASS2MLIR findings — ~20% to 100%+ GPU performance improvements across Ampere, Blackwell, and Jetson

We’re seeing up to 110% higher Qwen3.5 4B throughput on Jetson Orin Nano — benchmarks and repo available