"Zero-Shot Dense Retrieval with Embeddings from Relevance Feedback"

Smart document feedback replaces costly LLM generation in dense retrieval.

ReDE-RF uses real documents instead of generated ones for zero-shot dense retrieval.

📚 https://arxiv.org/abs/2410.21242

🎯 Original Problem:

Zero-shot dense retrieval systems struggle without relevance supervision data. Current methods like HyDE rely on LLMs to generate hypothetical documents, which has limitations in domain knowledge and efficiency.

-----

🛠️ Solution in this Paper:

→ Introduces Real Document Embeddings from Relevance Feedback (ReDE-RF)

→ Uses hybrid sparse-dense retrieval for initial document set

→ LLM judges relevance of retrieved documents with single-token output

→ Uses embeddings of relevant real documents to refine query representation

→ Falls back to hypothetical generation only when no relevant documents found

-----

💡 Key Insights:

→ Using real documents instead of generated ones ensures content is grounded in corpus

→ Single-token LLM output for relevance judgment is more efficient than document generation

→ Can be distilled into smaller model (DistillReDE) removing LLM dependency at inference

→ Hybrid retrieval (BM25 + dense) provides better initial document set

-----

📊 Results:

→ Surpasses state-of-the-art zero-shot methods by 6% in low-resource domains

→ Improves latency by 7.5-11.2x compared to context-based document generation

→ DistillReDE achieves 33% improvement over Contriever baseline

→ Maintains competitive performance in high-resource settings