u/NecessaryPay6108

I wanted to compare local code LLMs under real deployment conditions instead of relying on cloud benchmarks, so I evaluated 22 open-weight models on the MBPP benchmark using 11,000 automated inference runs.

Some observations:

- Qwen2.5-Coder-7B offered the best balance of accuracy and speed.

- Larger models didn't always perform better.

- Reasoning models incurred a significant latency cost without improving Pass@1 on MBPP.

- CodeBLEU and functional correctness often diverged.

I've published the benchmark methodology, results, and released the complete telemetry dataset on Hugging Face.

Medium: https://medium.com/@mrshahzebkhoso/11-000-inference-runs-22-slms-one-gpu-an-empirical-breakdown-of-local-code-generation-28efeb4f27ed

HF: https://huggingface.co/datasets/ShahzebKhoso/local-code-master\_telemetry\_arena