Smart document feedback replaces costly LLM generation in dense retrieval.
ReDE-RF uses real documents instead of generated ones for zero-shot dense retrieval.
📚 https://arxiv.org/abs/2410.21242
🎯 Original Problem:
Zero-shot dense retrieval systems struggle without relevance supervision data. Current methods like HyDE rely on LLMs to generate hypothetical documents, which has limitations in domain knowledge and efficiency.
-----
🛠️ Solution in this Paper:
→ Introduces Real Document Embeddings from Relevance Feedback (ReDE-RF)
→ Uses hybrid sparse-dense retrieval for initial document set
→ LLM judges relevance of retrieved documents with single-token output
→ Uses embeddings of relevant real documents to refine query representation
→ Falls back to hypothetical generation only when no relevant documents found
-----
💡 Key Insights:
→ Using real documents instead of generated ones ensures content is grounded in corpus
→ Single-token LLM output for relevance judgment is more efficient than document generation
→ Can be distilled into smaller model (DistillReDE) removing LLM dependency at inference
→ Hybrid retrieval (BM25 + dense) provides better initial document set
-----
📊 Results:
→ Surpasses state-of-the-art zero-shot methods by 6% in low-resource domains
→ Improves latency by 7.5-11.2x compared to context-based document generation
→ DistillReDE achieves 33% improvement over Contriever baseline
→ Maintains competitive performance in high-resource settings
Share this post