"Towards Learning to Reason: Comparing LLMs with Neuro-Symbolic on Arithmetic Relations in Abstract Reasoning"

LLMs stumble on math, while vector-symbolic AI aces abstract reasoning tests.

This paper compares LLMs with neuro-symbolic approaches in solving Raven's Progressive Matrices, revealing limitations in LLMs' arithmetic reasoning capabilities.

-----

https://arxiv.org/abs/2412.05586

🤔 Original Problem:

→ Current LLMs struggle with abstract reasoning tasks, particularly in understanding and executing arithmetic operations within Raven's Progressive Matrices (RPM).

→ Even with perfect visual perception input, LLMs fail to achieve complete accuracy in solving RPM puzzles.

-----

🔧 Solution in this Paper:

→ The researchers introduced Abductive Rule Learner with Context-awareness (ARLC), a neuro-symbolic approach using vector-symbolic architectures.

→ ARLC represents concepts as distributed vectors where dot products define similarity kernels.

→ Simple element-wise operations on vectors perform addition/subtraction on encoded values.

→ The system learns RPM rules as a differentiable assignment problem.

-----

💡 Key Insights:

→ LLMs show significant weakness in arithmetic operations, especially with larger matrices and expanded dynamic ranges

→ ARLC maintains high accuracy across different matrix sizes without retraining

→ Vector-symbolic architectures enable better handling of mathematical operations than traditional LLM approaches

-----

📊 Results:

→ GPT-4 achieved 93.2% accuracy on I-RAVEN dataset

→ ARLC achieved 98.4% accuracy on the same dataset

→ LLMs' accuracy drops below 10% on larger matrices with expanded dynamic ranges

→ ARLC maintains high accuracy even with 3×10 matrices and dynamic ranges up to 1000