"Steps are all you need: Rethinking STEM Education with Prompt Engineering"

Teaching LLMs physics through examples and step-by-step reasoning

This paper introduces specialized prompting techniques combining Chain-of-Thought and analogical reasoning to enhance physics problem-solving capabilities in open-source LLMs.

-----

https://arxiv.org/abs/2412.05023

🤔 Original Problem:

LLMs struggle with physics and mathematics problems due to limited mathematical reasoning abilities and tendency to hallucinate. Current prompting methods don't effectively address these limitations.

-----

🔧 Solution in this Paper:

→ Developed StemStep dataset containing 928 high-quality physics and mathematics questions with step-by-step solutions

→ Implemented K-Shot Chain-of-Thought prompting to break complex problems into manageable steps

→ Created Analogical Chain-of-Thought prompting, combining example-based reasoning with step-by-step problem solving

→ Evaluated performance on Mistral 7B and Mixtral 8x7B models using various prompting techniques

-----

💡 Key Insights:

→ Mixture of Experts architecture significantly improves physics problem-solving accuracy

→ Performance peaks at K=3 for Mistral 7B and K=6 for Mixtral 8x7B

→ Open-source models need domain-specific training for effective analogical reasoning

-----

📊 Results:

→ Mixtral 8x7B achieved 66.2% accuracy with Analogical CoT, up from 42% baseline

→ Mistral 7B showed 53% accuracy with K-Shot CoT, improving from 31.5% baseline

→ Combined CoT and Analogical prompting outperformed individual techniques