"TURBOATTENTION: Efficient Attention Approximation For High Throughputs LLMs"

Achieves 1.2-1.8x speedup in attention computation, by turning floating-point math into simple integer operations.

TurboAttention enables faster LLM inference by optimizing attention computation through quantization and compression while maintaining accuracy.

-----

https://arxiv.org/abs/2412.08585

Original Problem 🤔:

→ Current LLM inference faces major bottlenecks in attention computation, using high-precision formats that slow down processing and consume excessive memory

→ Existing solutions either focus solely on weight quantization or require floating-point operations, missing opportunities for comprehensive optimization

-----

Solution in this Paper 🔧:

→ TurboAttention introduces FlashQ, a technique that quantizes attention computation using mixed-precision (2-bit and 4-bit) formats

→ The system compresses Key-Value cache progressively, first to INT8 then to lower bits, optimizing both memory and speed

→ It implements Sparse Activated Softmax (SAS), replacing expensive floating-point operations with efficient integer calculations

→ The solution integrates with FlashAttention's tiling mechanism, ensuring compatibility with existing acceleration methods

-----

Key Insights 💡:

→ Different attention heads can tolerate different compression levels without accuracy loss

→ Exponential calculations in softmax can be approximated using lookup tables and polynomial functions

→ Progressive quantization balances computational efficiency with memory savings

-----

Results 📊:

→ Achieves 1.2-1.8x speedup in attention computation

→ Reduces KV cache size by 4.4x

→ Delivers 2.37x maximum throughput improvement over FP16 baseline

→ Maintains near-lossless accuracy across mathematical and symbolic reasoning tasks