MoRA (Mixture of Refinement Agents ) makes LLMs better physicists by fixing their reasoning errors one step at a time
This paper introduces Mixture of Refinement Agents (MoRA), a framework that enhances physics reasoning in LLMs by identifying and correcting three key error types: problem miscomprehension, incorrect concept application, and computational errors. MoRA uses specialized agents for iterative refinement, significantly improving open-source LLM performance.
-----
https://arxiv.org/abs/2412.00821
🤔 Original Problem:
LLMs struggle with physics reasoning tasks due to three main challenges: misunderstanding problems, applying wrong concepts, and making computational errors. Current solutions address these issues separately, lacking a unified approach.
-----
🔧 Solution in this Paper:
→ MoRA uses GPT-4o as an error identifier to analyze LLM solutions and assign specific flags and scores.
→ Three specialized refinement agents tackle different error types: miscomprehension, concept application, and computation.
→ The framework works iteratively, with each agent refining solutions until all errors are resolved or maximum iterations reached.
→ For concept refinement, MoRA uses GraphRAG to retrieve correct physics knowledge from an external database.
-----
💡 Key Insights:
→ Problem miscomprehension errors are least common but crucial to address first
→ Concept application errors require external knowledge base integration
→ Computational errors form the majority of mistakes in LLM solutions
-----
📊 Results:
→ MoRA improved Llama-3-70B accuracy by 13.38% on PhysicsQA benchmark
→ Enhanced Gemma-2-27B performance by 16.03% on complex physics problems
→ Achieved 64% prediction accuracy on test sets
