""Give Me BF16 or Give Me Death"? Accuracy-Performance Trade-Offs in LLM Quantization"

Paper reveals optimal quantization formats for different LLM deployment scenarios.

→ W8A8-FP quantization is essentially lossless across all model scales

https://arxiv.org/abs/2411.02355

🎯 Original Problem:

Uncertainty around accuracy-performance trade-offs in different LLM quantization formats creates barriers to adoption. Engineers need clear guidelines on which quantization method works best for their specific deployment scenarios.

-----

🔧 Solution in this Paper:

→ Evaluated three quantization formats across Llama-3.1 model family (8B, 70B, 405B):

- W8A8-FP: 8-bit floating point for weights/activations

- W8A8-INT: 8-bit integer for weights/activations

- W4A16-INT: 4-bit integer weights with 16-bit activations

→ Used comprehensive benchmarking:

- Academic benchmarks (Open LLM Leaderboard V1/V2)

- Real-world tasks (Arena-Hard-Auto, HumanEval)

- Text similarity analysis between quantized and uncompressed outputs

-----

💡 Key Insights:

→ W8A8-FP quantization is essentially lossless across all model scales

→ W8A8-INT shows minimal accuracy loss (1-3%) when properly tuned

→ W4A16-INT maintains competitive accuracy comparable to W8A8-INT

→ Larger quantized models closely follow original word choices and sentence structures

-----

📊 Results:

→ W4A16-INT performs best for synchronous deployments and small models on mid-tier GPUs

→ W8A8 formats excel for mid/large models on high-end GPUs in asynchronous deployment

→ Choice depends on model size, hardware, use case, and deployment scenario