Smart solver picks optimal LLM or symbolic method for each program synthesis task.
CYANEA dynamically selects between LLMs and symbolic solvers for program synthesis tasks, using a multi-armed bandit algorithm to optimize performance and cost while adapting to each unique problem.
-----
https://arxiv.org/abs/2501.05247
Original Problem 🤔:
→ Program synthesis using LLMs shows varying performance across different tasks and prompt styles. Users struggle to select the optimal LLM or solver for each synthesis task, leading to wasted resources and suboptimal results.
-----
Solution in this Paper 💡:
→ CYANEA frames solver selection as an online learning problem using multi-armed bandit algorithms.
→ The system maintains a portfolio of LLM-prompt pairs and symbolic solvers.
→ For each synthesis task, it predicts the most promising solver sequence.
→ The system uses k-Nearest Neighbor to learn from previous task performance.
→ CYANEA allocates computational budgets dynamically based on historical solver behavior.
-----
Key Insights from this Paper 🔍:
→ LLM performance varies significantly across different synthesis tasks
→ Combining symbolic solvers with LLMs outperforms either approach alone
→ Dynamic prompt selection improves synthesis success rates
→ Cost-aware resource allocation enhances efficiency
-----
Results 📊:
→ Solves 37.2% more queries than best single solver
→ Achieves results within 4% of virtual best solver
→ Single k-NN variant solves 88.3% of test queries
→ Outperforms baseline Par-2 scores by over 60%
Share this post