MathCoder2: Better Math Reasoning from Continued Pretraining on Model-translated Mathematical Code

This paper makes AI better at math by connecting Python code with step-by-step explanations.

Basically, teaching machines math by showing them both the code and the thinking process

MathCoder2 pipeline generates mathematical code with reasoning steps.

📚 https://arxiv.org/abs/2410.08196

Original Problem 🔍:

LLMs struggle with mathematical reasoning tasks due to limited exposure to high-quality mathematical content during pretraining.

-----

Solution in this Paper 🧠:

• Introduces MathCode-Pile: A 19.2B-token dataset for continued mathematical pretraining

• Generates mathematical code with corresponding reasoning steps using Llama-3.1-70B-Instruct

• Extracts LaTeX expressions, conditions, and results from math texts

• Translates extracted info into Python code snippets

• Executes code and verifies correctness

• Pairs verified code with original reasoning steps

• Combines web data, synthetic data, math code, and textbooks

-----

Key Insights from this Paper 💡:

• Pairing mathematical code with natural language reasoning enhances LLM performance

• Verifying generated code correctness improves dataset quality

• Open-sourcing the entire pipeline promotes transparency and reproducibility

• Continued pretraining on diverse mathematical content significantly boosts reasoning abilities

-----

Results 📊:

• MathCoder2-Llama-3-8B achieves 38.4% accuracy on MATH (17% improvement)

• 69.9% accuracy on GSM8K (15.1% improvement)

• Outperforms some closed-source math models of similar size

• Competitive results across five mathematical benchmarks

• 2.7B tokens of high-quality generated mathematical code with reasoning steps