"MAIN-RAG: Multi-Agent Filtering Retrieval-Augmented Generation"

MAIN-RAG lets LLMs clean up their own knowledge retrieval mess, making answers more reliable.

MAIN-RAG introduces a training-free multi-agent system where LLMs collaborate to filter and rank retrieved documents, improving RAG accuracy without additional training.

-----

https://arxiv.org/abs/2501.00332

🔍 Original Problem:

→ Traditional RAG systems struggle with noisy document retrieval, leading to decreased accuracy and increased computational overhead

→ Current solutions either require extensive training or are sensitive to input prompts

-----

🛠️ Solution in this Paper:

→ MAIN-RAG uses three specialized LLM agents working together: Predictor, Judge, and Final-Predictor

→ Agent-1 (Predictor) generates initial answers for each retrieved document

→ Agent-2 (Judge) evaluates document relevance using a novel scoring mechanism based on log probability differences

→ Agent-3 (Final-Predictor) generates the final answer using filtered and ranked documents

→ An adaptive judge bar dynamically adjusts filtering thresholds based on document score distributions

-----

💡 Key Insights:

→ Document ordering significantly impacts RAG performance

→ Related documents show high scores with low variance, while noisy documents have uniform distribution

→ Adaptive filtering thresholds perform better than fixed thresholds

-----

📊 Results:

→ 2-11% improvement in answer accuracy across 4 QA benchmarks

→ Outperforms training-free baselines by 6.1% with Mistral-7B

→ Matches performance of training-based methods without requiring additional compute

------

Are you into AI and LLMs❓ Join my daily AI newsletter. I will send you 7 emails a week analyzing the highest signal AI developments. ↓↓

🎉 https://rohanpaul.substack.com/