Proactive data curation with DATA ADVISOR, and integrating guiding principles to improve LLM data generation, achieving better safety and utility outcomes.
📚 https://arxiv.org/abs/2410.05269
Original Problem 🚨:
LLM-generated data often lacks quality, with underrepresented aspects and low-quality points, affecting model alignment with safety and utility goals.
Solution in this Paper 🛠:
- DATA ADVISOR uses predefined principles to guide LLM data generation.
- Monitors generated data, identifies weaknesses, advises next iterations.
- Integrates easily with existing methods like Self-Instruct.
- Focuses on dataset-level control for safety and diversity.
Key Insights from this Paper 💡:
- Dynamic guidance improves data quality and coverage.
- Enhances safety alignment without sacrificing utility.
- Versatile for broader applications like instruction tuning.
Results 📊:
- Safety scores improved by 10.1 on CatQA, 4.6 on BeaverTails.
- Utility scores increased by 1.6 on MMLU.
- Outperforms Self-Instruct in safety and utility across models.
Share this post