"Linear Correlation in LM's Compositional Generalization and Hallucination"

Below podcast on this paper is generated with Google's Illuminate.

https://arxiv.org/abs/2502.04520

The paper addresses the challenge of understanding how LLMs generalize knowledge, especially in compositional tasks. LLMs struggle with basic knowledge composition, like reverse or transition inference.

This paper proposes that linear correlations exist between related knowledge within LLMs. It suggests that a linear transformation can map next token prediction logits from one knowledge prompt to another, mirroring human knowledge composition.

-----

📌 This paper reveals a critical insight: LLMs implicitly use linear transformations for knowledge composition. This linearity, observed in logit space, explains both generalization and hallucination as a function of transformation precision.

📌 The resilient linear correlation, even after fine-tuning, suggests a fundamental architectural bias in LLMs. This bias, rooted in vocabulary representations, predetermines how knowledge is composed and generalized.

📌 The linear transformation matrix W acts as a knowledge composition operator. Its weights directly reflect real-world correlations and its precision governs the accuracy of knowledge transfer across related prompts.

----------

Methods Explored in this Paper 🔧:

→ This paper investigates linear correlations in LLMs during knowledge composition.

→ The authors fit a linear transformation (W, b) between the next token prediction logits of related prompts. For example, they examined the relationship between "X lives in the city of" and "X lives in the country of".

→ They sampled numerous output logits from prompts with various inputs to fit this transformation.

→ Pearson correlation coefficients were used to evaluate the linear relationship between different knowledge types.

→ The precision of the linear transformation W was analyzed by checking if its weights align with real-world knowledge pairs, using Hit@Top-N metrics.

-----

Key Insights 💡:

→ A linear transformation can approximate the relationship between next token prediction logits for related knowledge.

→ This linear correlation is resilient to large-scale fine-tuning of LLMs.

→ The linear transformation's weights often mirror real-world knowledge relationships. High weights are assigned to correct knowledge pairs like (Paris, France) in the City→Country example.

→ When the linear transformation is precise, it can enable compositional generalization.

→ However, an imprecise linear transformation can lead to compositional hallucination, where learning one knowledge incorrectly generalizes to related knowledge.

→ Vocabulary representations are crucial for forming these linear correlations and enabling knowledge composition.

-----

Results 📊:

→ High Pearson correlation coefficients were observed for related knowledge pairs like City→Country (0.89) and Math operations (0.93 for X+1→X+2).

→ Lower correlation was found for less related pairs like CEO→Company (0.55) and cross-language pairs.

→ In City→Country knowledge composition, W precision reached 42% for top-1 influenced cities and 67% for top-1 influencing countries using Hit@Top-1 metric.

→ Generalization success was significant (53.70% for City→Country) only when both correlation intensity and W precision were high.