Smart routing between specialized embedding experts beats one-size-fits-all search models.
The paper finds routing among domain-specific embeddings yields superior retrieval performance over single models.
📚 https://arxiv.org/pdf/2409.02685
Original Problem 🔍:
Information retrieval methods often rely on a single embedding model trained on large, general-domain datasets, limiting performance across diverse domains.
-----
Key Insights from this Paper 💡:
• Multiple domain-specific expert embedding models outperform single general-purpose models
• Effective routing between experts is crucial for leveraging domain-specific knowledge
• Benefits generalize to datasets without corresponding experts
• Parametric knowledge influences embedding extraction quality
• Adding diverse experts improves performance more than adding experts within the same domain
-----
Solution in this Paper 🛠️:
• ROUTERRETRIEVER: A retrieval model with multiple domain-specific experts and a routing mechanism
• Base encoder (Contriever) with LoRA-trained domain-specific gates
• Pilot embedding library for efficient routing
• Selects most appropriate expert for each query using similarity to pilot embeddings
• Lightweight and flexible - allows easy addition/removal of experts without retraining
-----
Results 📊:
• Outperforms MSMARCO-trained model by +2.1 nDCG@10 on BEIR benchmark
• Surpasses multi-task trained model by +3.2 nDCG@10
• Routing mechanism outperforms other common techniques by +1.8 on average
• Benefits generalize to datasets without specific experts
• Performance improves as more diverse experts are added
Share this post