"Don't Do RAG: When Cache-Augmented Generation is All You Need for Knowledge Tasks"

Precomputed key-value caches make knowledge retrieval 40x faster than traditional RAG.

Cache-augmented generation replaces traditional retrieval-augmented generation by preloading documents and precomputing key-value caches, making knowledge tasks faster and more accurate.

-----

https://arxiv.org/abs/2412.15605

🤔 Original Problem:

Traditional RAG systems suffer from retrieval latency, errors in document selection, and complex system architecture that requires careful tuning and maintenance.

-----

🔧 Solution in this Paper:

→ The paper introduces Cache-Augmented Generation (CAG), which preloads all relevant documents into LLM's memory before inference.

→ CAG precomputes key-value caches from documents, storing them for future use rather than retrieving during runtime.

→ The system operates in three phases: external knowledge preloading, inference with cached context, and efficient cache reset.

-----

💡 Key Insights:

→ Eliminating retrieval during inference dramatically reduces response time and system complexity.

→ Preloading context enables holistic understanding across all documents.

→ CAG works best when document collections fit within LLM context windows.

-----

📊 Results:

→ CAG achieves highest BERT-Score (0.7759) on HotPotQA, outperforming both sparse and dense RAG systems.

→ Generation time reduced from 94.34s to 2.32s on large datasets.

→ Consistent performance improvement across both SQuAD and HotPotQA benchmarks.