"MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMs"

Below podcast on this paper is generated with Google's Illuminate.

https://arxiv.org/abs/2501.17399

The paper addresses the challenge of evaluating LLMs in realistic multi-turn conversations, as existing benchmarks are insufficient for frontier models. It introduces MultiChallenge, a new benchmark to assess instruction following, context handling, and reasoning in conversations.

This paper proposes MultiChallenge, a benchmark with four challenge categories, and an automatic evaluation method using LLMs as judges with instance-level rubrics.

-----

📌 MultiChallenge pinpoints specific LLM failings in multi-turn context via targeted challenge categories, unlike superficial benchmarks. This enables granular performance analysis.

📌 Hybrid data generation, merging LLM agents with human review, offers a scalable method to create realistic and challenging conversational benchmarks.

📌 Instance-level rubric auto-evaluation achieves high human alignment, enabling reliable and efficient assessment of complex conversational LLM behaviors.

----------

Methods Explored in this Paper 🔧:

→ The paper introduces MultiChallenge, a novel benchmark designed to evaluate LLMs in multi-turn conversations.

→ MultiChallenge focuses on four realistic challenges: instruction retention, inference memory, reliable versioned editing, and self-coherence.

→ These challenges assess LLMs' ability to follow instructions, manage context, and reason within multi-turn dialogues.

→ The benchmark employs a hybrid approach for data creation, combining synthetic generation with human expert review and editing.

→ A multi-agent system named MMSE is used for synthetic data generation, incorporating topic seeds, personas, and challenge-specific configurations.

→ To enable automatic evaluation, the paper develops an LLM-as-judge system using instance-level rubrics.

→ Human raters create binary rubric questions for each test case, which are then used by LLM judges for automatic assessment.

-----

Key Insights 💡:

→ Current frontier LLMs, despite excelling in existing benchmarks, struggle with MultiChallenge, achieving less than 50% accuracy.

→ The four challenge categories in MultiChallenge effectively target distinct weaknesses in even the most advanced LLMs.

→ Instance-level rubrics significantly improve the alignment of automatic LLM evaluations with human ratings, reaching 93% alignment.

→ The hybrid data generation approach reduces human effort in benchmark creation while maintaining data quality and challenge.

-----

Results 📊:

→ Claude 3.5 Sonnet achieves the highest average accuracy of 41.4% on MultiChallenge among frontier models.

→ o1-preview achieves 37.23% average accuracy, outperforming other models except Claude 3.5 Sonnet.

→ Automatic evaluation with instance-level rubrics achieves 93.95% alignment with human raters, compared to 37.33% for baseline auto-evaluation.