"EscapeBench: Pushing Language Models to Think Outside the Box"

EscapeBench tests AI's ability to think outside the box through escape room puzzles

EscapeBench introduces a creative reasoning benchmark using escape room scenarios to test LLMs' ability to think innovatively and solve complex puzzles.

-----

https://arxiv.org/abs/2412.13549v1

🤔 Original Problem:

→ Current LLM benchmarks focus on explicit goal-oriented tasks, missing evaluation of creative problem-solving abilities

→ Models struggle with unconventional tool use and innovative thinking in unfamiliar environments

-----

🛠️ Solution in this Paper:

→ EscapeBench creates room escape game environments requiring creative reasoning and tool use

→ Players must discover implicit goals through exploration and trial-and-error

→ The framework includes Scenes (connected spaces), Tools (collectible objects), and Items (interactive elements)

→ EscapeAgent enhances model performance through Foresight (creative tool use) and Reflection (task tracking) modules

-----

💡 Key Insights:

→ Models heavily rely on conventional thinking patterns, struggling with creative adaptation

→ Input and Craft actions prove most challenging, showing models' limitations in creative tool combinations

→ Larger models benefit more from creative reasoning frameworks than smaller ones

-----

📊 Results:

→ Base models achieve only 15% average progress without hints

→ EscapeAgent reduces hint usage by 40% and requires fewer steps

→ Even best models need twice as many steps as human players

→ Performance gap widens in harder difficulty settings

------

Are you into AI and LLMs❓ Join my daily AI newsletter. I will send you 7 emails a week analyzing the highest signal AI developments. ↓↓

🎉 https://rohanpaul.substack.com/