"Critique Fine-Tuning: Learning to Critique is More Effective than Learning to Imitate"

The paper introduces Critique Fine-Tuning method. It proposes that training LLMs to critique responses is more effective than training them to imitate. This method enhances reasoning in LLMs, especially for complex tasks like mathematical problem-solving.

-----

📌 Training a LLM to critique instead of imitate forces it to engage in deeper reasoning. This shifts the focus from surface-level pattern recognition to identifying logical inconsistencies, improving generalization in complex reasoning tasks.

📌 Critique Fine-Tuning improves sample efficiency. The model achieves state-of-the-art performance with only 50,000 samples, compared to millions in standard Supervised Fine-Tuning. This suggests that structured critical thinking is a more efficient learning signal than direct answer imitation.

📌 Fine-tuning on critiques makes the model robust to noisy inputs. Instead of blindly following patterns, it learns to detect and correct errors. This can enhance reliability in mathematical problem-solving and other reasoning-intensive applications.

-----

https://arxiv.org/abs/2501.17703

Original Problem 🧐:

→ Supervised Fine-Tuning (SFT) is the standard method for training LLMs.

→ SFT trains LLMs to imitate annotated correct responses.

→ However, simply imitating responses might not deeply improve reasoning capabilities in LLMs, especially for already strong base models.

-----

Solution in this Paper 💡:

→ It trains LLMs to critique noisy responses instead of just imitating correct ones.

→ The training data for this method consists of (query, noisy response, critique) triplets.

→ The LLM is trained to predict the critique given a query and a noisy response.

→ This critique learning process encourages deeper analysis and critical thinking in LLMs.

→ The paper uses GPT-4o to generate critiques for a dataset of 50K samples from WebInstruct.

-----

Key Insights from this Paper 🧠:

→ Critique Fine-Tuning method improves LLM reasoning more effectively than standard SFT.

→ is particularly beneficial for mathematical reasoning tasks.

→ Is data-efficient, achieving strong performance with significantly fewer training samples than SFT.

→ Robust to different sources of noisy responses and teacher critique models.

→ Learning to critique helps LLMs develop a more nuanced understanding compared to simply imitating solutions.