Smart query generation helps LLMs pick perfect tools for complex tasks.
This paper introduces a method to improve tool retrieval for LLMs by generating optimized search queries. Instead of directly matching user requests with tool descriptions, it leverages LLMs to generate contextually-aware queries that better capture the user's intent and required functionality.
----
https://arxiv.org/abs/2412.03573
🤔 Original Problem:
Current tool retrieval methods like BM25 and dense embeddings lack contextual understanding needed for complex user requests. Simple matching often misses relevant tools or gets misled by irrelevant information.
-----
🔧 Solution in this Paper:
→ The paper proposes using LLMs to dynamically generate tool retrieval queries based on user utterances.
→ Each generated query describes a specific tool needed to accomplish the request.
→ Three approaches are explored: zero-shot prompting, supervised fine-tuning, and alignment learning.
→ The alignment learning method iteratively optimizes query generation by measuring retrieval performance.
-----
💡 Key Insights:
→ Adding original user utterance to generated queries improves retrieval performance
→ Alignment learning performs best for out-of-domain (unseen) tools
→ Supervised fine-tuning works better for in-domain (seen) tools
-----
📊 Results:
→ Alignment learning improved out-of-domain Recall@5 by 78.53%
→ Supervised fine-tuning achieved 87.29% Recall@5 for in-domain tools
→ Both methods significantly outperformed baseline utterance-only retrieval
Share this post