CUDA/LLM engineers: would you actually use a configurable Llama runtime?
I'm building a CUDA-native LLM runtime specifically for experimenting with GPU-level optimization on consumer GPUs, and I'd like some feedback.
The idea is basically a hackable Llama runtime where you can actually get into the CUDA kernels instead of fighting through a massive production inference stack.
The runtime is intended to let developers/researchers:
* Modify GEMM / Tensor Core kernels
* Experiment with FlashAttention and PagedAttention
* Tune KV-cache behavior
* Change tile sizes, memory layouts and thread configurations
* Experiment with kernel fusion and asynchronous execution
* Profile the resulting kernels
* Tune the runtime around the actual GPU they're running on
I'm doing this as my final-year engineering project, and I'm trying to determine whether this is actually useful to people who work with CUDA/LLMs/local inference.
2–3 minute survey:
https://forms.gle/KM4fUzVY1oC7g4TP8
If you've worked with CUDA, LLM inference, GPU optimization, llama.cpp, vLLM, TensorRT-LLM, FlashAttention, etc., I'd particularly appreciate your input.