"Debate Helps Weak-to-Strong Generalization"

Leveraging model debates enhances weak-to-strong generalization, closing the performance gap in LLM alignment.

This paper addresses the challenge of aligning increasingly powerful LLMs when human supervision becomes weak. It proposes using debate between strong models to enhance weak supervision and improve weak-to-strong generalization.

-----

Paper - https://arxiv.org/abs/2501.13124

Original Problem 🤔:

→ Current AI alignment relies on high quality human supervision which becomes insufficient for future superhuman models.

→ This limitation weakens the effectiveness of existing alignment approaches and could compromise AI safety.

→ Scalable oversight and weak-to-strong generalization are approaches to handle this issue, but are often studied separately.

-----

Solution in this Paper 💡:

→ This paper combines scalable oversight and weak-to-strong generalization.

→ It uses a strong pretrained model to improve weak model supervision.

→ Then it uses this enhanced weak supervision to train the strong model.

→ The method employs debate between two instances of a strong model to extract trustworthy information.

→ In debate, two strong models argue for opposing answers, making it harder to lie.

→ Arguments from debate inform a weak model, providing contextual information for training.

→ An ensemble of weak models is trained to process long debate arguments robustly.

→ Predictions from the weak model ensemble are used as supervision to fine-tune a strong student model.

-----

Key Insights from this Paper 🧐:

→ Debate assists weak models in extracting reliable information from strong, potentially untrustworthy models.

→ Debate provides valuable context for training weak models, improving their supervisory capability.

→ An ensemble of weak models effectively exploits long arguments from strong model debates.

→ Debate ensembles, differing in debate sampling seeds, outperform finetune ensembles and single weak models.

-----

Results 📈:

→ Achieves up to 76.5% Performance Gap Recovered (PGR) on SciQ, significantly outperforming baselines.

→ Reaches 69.2% PGR on BoolQ, 56.5% PGR on CosmosQA and 70.0% PGR on AnthropicHH, consistently surpassing baselines.

→ Debate ensembles show better performance than single weak models and finetune ensembles across tasks.