Multiple AI experts working together make better decisions than a single expert
https://arxiv.org/abs/2411.00492
🤖 Original Problem:
Single-expert prompting frameworks like ExpertPrompting can introduce bias and limit perspectives when handling open-ended queries. They often provide one-sided views, missing the depth needed for comprehensive answers.
-----
🛠️ Solution in this Paper:
→ Introduces Multi-expert Prompting with two main steps:
The method has two main steps:
1. Expert & Response Generation - Generates n expert identities with concise role descriptions and gets responses from each expert
2. Expert Response Aggregation - Uses 7 carefully designed subtasks based on Nominal Group Technique to combine expert responses and select the best output.
→ Uses zero-shot prompting to generate diverse expert responses in parallel
→ Implements a novel 7-subtask method to:
- Identify agreed viewpoints
- Handle conflicting opinions
- Capture unique perspectives
- Select best response through systematic evaluation
-----
💡 Key Insights:
→ Three experts yield optimal results for truthfulness and factuality
→ Aggregating expert responses in a single turn is more efficient than iterative refinement
→ Human-designed NGT framework outperforms LLM-generated plans for response aggregation
→ Diverse domain experts working in parallel produce better results than sequential expert consultation
-----
📊 Results:
→ Improves truthfulness by 8.69% with ChatGPT over best baseline
→ Achieves state-of-the-art truthfulness scores on TruthfulQA-Generation
→ Wins 75% cases for informativeness and 76.5% for usefulness
→ Completely eliminates toxic content and reduces hurtfulness
Share this post