"Anchor Attention, Small Cache: Code Generation with Large Language Models"

Discover how AnchorCoder trims KV caching by 95% without losing code generation prowess.

The paper introduces AnchorCoder, a novel method to reduce the computational load of Key-Value (KV) caching in LLMs used for code generation. This approach maintains model performance while significantly compressing contextual information.

-----

https://arxiv.org/abs/2411.06680

Original Problem 🤔:

LLMs face high computational demands due to extensive KV caching, which is especially problematic in code generation tasks. Existing compression methods for natural language processing (NLP) don't work well for code.

-----

Solution in this Paper 💡:

→ AnchorCoder introduces a token-wise anchor attention mechanism that efficiently compresses contextual information.

→ This mechanism identifies key tokens (anchors) that capture essential context, reducing the need for full KV caching.

→ Layer-wise anchor attention ensures that information is preserved as it moves through the model's layers.

→ The system is designed to work across various model sizes, from 0.1 billion to 7 billion parameters, maintaining efficiency and performance.

-----

Key Insights from this Paper 📚:

→ Sparsity patterns in attention weights are crucial for efficient compression.

→ Token-wise and layer-wise anchor attention mechanisms are effective in reducing KV cache size.

→ The method balances compression with performance retention across different model sizes.

-----

Results 📈:

→ AnchorCoder achieves up to 95% reduction in KV cache size without degrading performance on benchmarks like HumanEval and MBPP.

→ Performance remains consistent when trained from scratch compared to dense attention models.