"Ranking Unraveled: Recipes for LLM Rankings in Head-to-Head AI Combat"

A deep dive into how different LLMs stack up against each other in head-to-head battles, revealing which ranking system works best

This paper evaluates different ranking systems (Elo, Bradley-Terry, Glicko, Markov Chain) for comparing LLMs through head-to-head battles. It reveals key insights about ranking stability, transitivity, and prediction accuracy across different evaluation scenarios.

-----

https://arxiv.org/abs/2411.14483

🤔 Original Problem:

→ Current LLM evaluation methods using benchmarks like GLUE fail to capture nuanced performance in complex tasks

→ Head-to-head comparisons between models need reliable ranking systems, but existing methods like Elo have limitations

-----

🔧 Solution in this Paper:

→ The paper introduces a systematic framework to evaluate ranking algorithms for LLMs

→ It analyzes four ranking systems: Elo, Bradley-Terry, Glicko, and Markov Chain

→ The study examines two evaluation scenarios: Arena Style (dynamic, uneven matchups) and Controlled Style (balanced matchups)

→ Key properties assessed include transitivity preservation, prediction accuracy, and sensitivity to hyperparameters

-----

💡 Key Insights:

→ Bradley-Terry model preserves transitivity better than other methods

→ Ranking systems perform differently in Arena vs Controlled evaluation styles

→ Hyperparameter sensitivity significantly impacts ranking reliability

-----

📊 Results:

→ Bradley-Terry achieves 77.29% transitivity preservation on Arena dataset

→ Elo shows highest prediction accuracy with 0.90 F1 score

→ Glicko demonstrates most stable performance across different hyperparameters