![[Paper] Statistically-Lossless Quantization of Large Language Models](https://preview.redd.it/4u0iz2l5t7fh1.png?auto=webp&s=c3efe50cd39bc85e790caa9df01c2cbda37f126a)
[Paper] Statistically-Lossless Quantization of Large Language Models
>Model quantization has become essential for efficient large language model deployment, yet existing approaches involve clear trade-offs: methods such as GPTQ and AWQ achieve practical compression but are lossy, while lossless techniques preserve fidelity but typically do not accelerate inference. This paper explores the middle ground of statistically-lossless compression through three complementary notions of losslessness for quantized LLMs. First, task-lossless compression preserves zero-shot benchmark accuracy within natural sampling variance and remains achievable at aggressive bitwidths. Second, we formalize the stricter notion of distribution-lossless compression, requiring the quantized model's next-token distribution to be practically indistinguishable from the original, and propose the Expected Acceptance Rate (EAR), the maximum token-agreement probability under optimal coupling, as a directly interpretable fidelity metric (for example, EAR >= 0.99 indicates 99% agreement). Third, we prove a gamma-squared variance law showing that symmetric quantization inflates noise variance by gamma squared relative to asymmetric quantization, making asymmetry necessary for distribution-lossless fidelity but not for task-level preservation. Using SLQ, a layer-wise non-uniform method with asymmetric quantization and wide bitwidth search, we achieve task-lossless compression at well below 4 bits per parameter (as low as 3.3 bits depending on the model), distribution-lossless compression at 5 to 6 bits per parameter on average, and inference speedups of 1.7 to 3.6x relative to FP16 with optimized kernels.
- arXiv : https://arxiv.org/abs/2605.02404
- Full Paper : https://arxiv.org/pdf/2605.02404
- GitHub : https://github.com/IST-DASLab/SLQ (Code coming soon)
Note : This is 2 Months old Paper & Repo. Sharing this as RedHat AI tweeted this sometime back.
In Full Paper, I found llama.cpp & GG few times.
>From the accuracy/compression perspective, existing approaches can be clustered into two categories. The first is represented by lossy compression techniques, such as Roundto-Nearest (RTN) quantization (Dettmers et al., 2022), llama.cpp (Gerganov & llama.cpp contributors, 2023), GPTQ (Frantar et al., 2023), or AWQ (Lin et al., 2024) which seek to map existing models to popular hardware-supported formats, such as 4-bit grouped weight quantization.
>We focus on obtaining near-lossless quantized models via layer-wise non-uniform scalar quantization, chosen for its broad support across GPUs (Frantar et al., 2024; 2023; Lin et al., 2024) and CPUs (Gerganov & llama.cpp contributors, 2023; Pegolotti et al., 2023; Ma et al., 2024);