Humans can integrate fragmented knowledge to form logical conclusions.
This paper investigates whether Transformer models can do the same by introducing the FTCT (Fragmented at Training, Chained at Testing) dataset.
It shows that few-shot Chain-of-Thought (CoT) prompting significantly improves Transformers' ability to perform compositional reasoning by reconstructing logical chains from fragmented data.
📌 Transformers are learning latent programs. The model is not just memorizing examples but extracting an implicit algorithmic structure from the fragmented data.
📌 Induction heads act as knowledge linkers. The way attention heads retrieve parent relationships resembles gradient descent in function space, explaining in-context learning behavior.
📌 Data structure is key to generalization. The phase transition at a knowledge ratio of 0.3 suggests that reasoning ability is not continuous but emerges abruptly, reinforcing the importance of dataset design.
-----
Paper - https://arxiv.org/abs/2501.15857
Original Problem 🤔:
→ Humans can deduce relationships between concepts that were never explicitly linked in training.
→ Traditional evaluation of compositional reasoning in Transformers is difficult due to the complexity of real-world datasets.
→ The paper asks: Can Transformers integrate separate pieces of learned knowledge and generalize to unseen reasoning chains?
-----
Solution in this Paper 🔧:
→ The FTCT dataset is introduced, where training data consists of short, disconnected knowledge fragments, while testing requires reconstructing full reasoning chains.
→ Transformers perform poorly in zero-shot settings but significantly improve with few-shot CoT prompting, which provides the correct vertex order for reasoning.
→ Compositional reasoning ability emerges when training-testing data similarity (measured by the relative knowledge ratio) exceeds 0.3.
→ Multi-layer attention mechanisms are crucial for this reasoning ability. Single-layer Transformers fail, while models with at least 2 layers and 2 heads succeed.
→ Transformers simulate an underlying generalizable program, utilizing induction heads and attention assignment to integrate fragmented knowledge into logical chains.
-----
Key Insights 💡:
→ Few-shot CoT prompting enables compositional reasoning, improving performance over zero-shot cases by revealing correct reasoning paths.
→ Higher data similarity between training and testing improves reasoning performance. A critical phase transition occurs when the relative knowledge ratio ≥ 0.3.
→ Multi-layer Transformers outperform simple models. Induction heads and attention mechanisms play a key role in in-context learning and retrieving parent relationships.
-----
Results 📊:
→ Few-shot CoT prompting significantly improves performance, boosting whole-chain accuracy from near 0% (zero-shot) to over 99% (few-shot).
→ Testing accuracy sharply increases when the relative knowledge ratio reaches 0.3, showing a clear threshold for compositional reasoning ability.
→ Transformers with at least 2 layers and 2 heads achieve near-optimal reasoning performance, while single-layer models fail.
Share this post