"ICLR: In-Context Learning of Representations"

LLMs don't just memorize word meanings, they can relearn them based on new examples

LLMs can learn new meanings for words through in-context examples, reshaping their internal representations to match the new context-specific semantics.

-----

https://arxiv.org/abs/2501.00070

🤔 Original Problem:

→ LLMs have fixed semantic representations learned during pretraining, but need flexibility to handle novel meanings in different contexts

-----

🔧 Solution in this Paper:

→ Designed a "graph tracing" task where familiar words (like apple, bird) are arranged in specific graph structures (grid, ring, hexagonal)

→ Fed the model with examples of random walks on these graphs to see if it can learn new relationships between words

→ Analyzed how internal representations reorganize using Principal Component Analysis and Dirichlet energy measurements

→ Used Llama-3.1-8B as primary model, with additional experiments on other variants

-----

💡 Key Insights:

→ LLMs can dynamically alter their pretrained word representations based purely on in-context examples

→ Representation reorganization happens suddenly at a critical context length, following power-law scaling

→ When words have strong semantic links (like days of week), new structure emerges in higher dimensions without fully overriding original meanings

→ Process follows energy minimization patterns similar to physical systems

-----

📊 Results:

→ Model achieves high accuracy (>80%) in predicting valid next nodes after seeing sufficient examples

→ Dirichlet energy of representations decreases before accuracy improvements, indicating structure learning

→ Performance scales with graph size following power-law relationship

------

Are you into AI and LLMs❓ Join me on X/Twitter with 52K+ others, to remain on the bleeding-edge of AI every day.

𝕏/🐦 https://x.com/rohanpaul_ai