This model reads LLM's thought process to predict if it's heading towards correct answers.
Lachesis predicts if LLM reasoning paths will lead to correct answers by analyzing structural patterns in multiple inference paths, enabling early detection of likely incorrect answers.
-----
https://arxiv.org/abs/2412.08281
🤔 Original Problem:
Self-consistency in LLMs requires multiple inference runs to validate answers, making it computationally expensive. No way exists to predict if these runs will yield correct results before completion.
-----
🔍 Solution in this Paper:
→ Lachesis converts reasoning paths into two formats: LLM Inference Matrix (LIM) and LLM Inference Graph (LIG)
→ LIM represents multiple reasoning paths as matrix columns, with each row showing function calls
→ LIG creates a directed graph where nodes are reasoning steps and edges show their connections
→ The model uses LSTM and GCN networks to process these representations
→ Four embedding types are used: Shape Only, Function Type Only, Function Type with Arguments, and Full embedding including answers
-----
💡 Key Insights:
→ Multiple reasoning paths converging to same answer indicates higher accuracy
→ Graph structure of reasoning paths reveals prediction patterns
→ Adding function arguments and answer information improves prediction accuracy
-----
📊 Results:
→ GCN model achieved 0.8136 precision with full embedding
→ Best accuracy: 0.7454 using GCN
→ Highest recall: 0.9182 with Shape Only embedding
Share this post