"HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMs"

Teaching LLMs to think like doctors through step-by-step medical reasoning verification

HuatuoGPT-o1 enhances medical reasoning in LLMs by using verifiable medical problems and a two-stage approach combining search strategies with reinforcement learning.

-----

https://arxiv.org/abs/2412.18925

🤔 Original Problem:

→ While LLMs show strong mathematical reasoning, medical reasoning remains underexplored despite its critical importance in healthcare

→ Verifying medical reasoning is challenging unlike mathematics, where solutions can be easily checked

-----

🔬 Solution in this Paper:

→ Created 40K verifiable medical problems from exam questions with clear ground-truth answers

→ Developed a medical verifier using GPT-4o to check solution correctness

→ Implemented a two-stage training approach: First, using search strategies (Backtracking, Exploring New Paths, Verification, Correction) to find complex reasoning paths for fine-tuning

→ Second, applying reinforcement learning with verifier-based rewards to enhance reasoning capabilities

-----

🎯 Key Insights:

→ Complex reasoning significantly improves medical problem-solving compared to simple approaches

→ Longer reasoning paths (712 tokens) provide richer feedback for reinforcement learning

→ Method successfully adapts to other domains like Chinese medical reasoning

-----

📊 Results:

→ HuatuoGPT-o1-8B showed 8.5-point improvement on medical benchmarks

→ 70B version outperformed other open-source medical LLMs across all benchmarks

→ Achieved 96.5% verification accuracy in Stage 1 and 94.5% in Stage 2