Teaching AI to explain itself - turns out it's pretty good at it!
LLMs can generate high-quality explanations for Natural Language Inference tasks that match human explanations in effectiveness while reducing annotation costs.
-----
https://arxiv.org/abs/2412.13942
Original Problem 🤔:
Collecting human explanations for Natural Language Inference (NLI) tasks is expensive and time-consuming. While LLMs can approximate human judgment distributions using explanations, they still require human-written explanations.
-----
Solution in this Paper 🛠️:
→ The paper proposes using LLM-generated explanations instead of human explanations for NLI tasks
→ It introduces two strategies: Label-Free (one explanation per NLI label) and Label-Guided (explanations based on human annotations)
→ The system leverages multiple LLMs like Llama3-Chat-70b and GPT4 to generate diverse explanations
→ These explanations are combined with a few human labels to approximate human judgment distributions
-----
Key Insights 💡:
→ LLM explanations perform similarly to human explanations when guided by human labels
→ The method generalizes well to datasets without explanations
→ Explanation variability serves as an indicator for human label variation
→ The approach works effectively on out-of-distribution test sets
-----
Results 📊:
→ Achieves comparable KL divergence (0.234) to human explanations (0.238)
→ Shows robust performance on ANLI dataset with 35.1% F1 score
→ Demonstrates successful generalization to MNLI dataset without explanations
→ Maintains effectiveness across different LLM architectures
Share this post