Not all powerful LLMs are good at generating training data - here's how to measure it.
This paper introduces AgoraBENCH, a comprehensive benchmark for evaluating how well LLMs generate synthetic training data. Through extensive experiments generating 1.26 million training instances using 6 LLMs and training 99 student models, it reveals distinct strengths among different LLMs in data generation capabilities.
-----
https://arxiv.org/abs/2412.03679
Original Problem 🤔:
While synthetic data generation using LLMs is becoming crucial for model training, there's no standardized way to evaluate different LLMs' abilities as data generators. Current research focuses on developing generation methods rather than comparing LLMs' generation capabilities.
-----
Solution in this Paper 🛠️:
→ AgoraBENCH provides standardized settings and metrics across nine configurations, combining three domains (math, instruction-following, code) with three data generation methods.
→ It introduces Performance Gap Recovered (PGR) metric to measure relative improvement of models trained on synthetic data.
→ The benchmark evaluates 6 LLMs including GPT-4o, Claude-3.5-Sonnet, and Llama-3.1 variants.
-----
Key Insights from this Paper 💡:
→ Different LLMs show distinct strengths - GPT-4o excels at generating new problems while Claude-3.5-Sonnet performs better at enhancing existing ones
→ An LLM's data generation ability doesn't necessarily correlate with its problem-solving capability
→ Multiple intrinsic features collectively indicate data generation quality
-----
Results 📊:
→ GPT-4o achieves highest PGR in 5 out of 9 settings
→ Top-5 principal components explain 93.4% variance in PGR values
→ JSON format shows 4.45% lower performance compared to free-form generation
Share this post