"Efficiently Learning at Test-Time: Active Fine-Tuning of LLMs"

SIFT (Selects Informative data for Fine-Tuning) makes LLMs learn better by picking the right training examples, not just the closest ones

This paper introduces SIFT (Selects Informative data for Fine-Tuning), an algorithm that improves test-time fine-tuning of LLMs by intelligently selecting relevant, non-redundant training data. Unlike traditional Nearest Neighbor retrieval, SIFT optimizes information gain while balancing relevance and diversity.

-----

https://arxiv.org/abs/2410.08020v1

🤔 Original Problem:

→ Current test-time fine-tuning methods rely on Nearest Neighbor retrieval, which often selects redundant data, limiting effectiveness

→ Existing approaches use fixed compute regardless of prompt complexity, leading to inefficient resource usage

-----

🔧 Solution in this Paper:

→ SIFT estimates uncertainty about model's response to a given prompt after fine-tuning on selected data

→ It selects data points that minimize this uncertainty while balancing relevance and diversity

→ The algorithm unifies ideas from retrieval and active learning to optimize overall information gain

→ SIFT adaptively invests compute based on expected performance improvements

-----

💡 Key Insights:

→ Uncertainty estimates can predict performance gains from fine-tuning

→ Test-time fine-tuning outperforms in-context learning with less computational overhead

→ SIFT naturally handles information duplication in training data

-----

📊 Results:

→ SIFT consistently outperforms Nearest Neighbor retrieval across all tested models

→ Achieves 5.1% better bits-per-byte metric compared to baselines

→ Minimal computational overhead (1.05x) compared to Nearest Neighbor retrieval