"Predicting Emergent Capabilities by Finetuning"

Predict tomorrow's AI capabilities by fine-tuning today's smaller models.

i.e. fine-tune weak models today to see what strong models will do tomorrow.

This paper introduces a method to predict when future LLMs will develop new capabilities, even when current models show random performance. The technique uses finetuning on smaller models to forecast emergence points in larger models, enabling early capability prediction up to 4x compute in advance.

-----

https://arxiv.org/abs/2411.16035

🎯 Original Problem:

While LLM pretraining loss follows predictable patterns, downstream capabilities often emerge suddenly and unpredictably. This creates challenges for developers, policymakers, and stakeholders who need to anticipate future model capabilities.

-----

🔍 Solution in this Paper:

→ The paper discovers that finetuning LLMs on specific tasks shifts the emergence point toward less capable models

→ By varying the amount of finetuning data, researchers can systematically track how emergence points shift

→ This insight enables fitting a parametric function ("emergence law") to predict when capabilities will emerge in future models

→ The method uses only small-scale pre-emergence models to predict capabilities in larger models

-----

💡 Key Insights:

→ Task-specific finetuning systematically shifts emergence points toward weaker models

→ The amount of finetuning data directly correlates with the magnitude of emergence shift

→ Emergence can be predicted using models with only 1/4th the compute needed for actual emergence

-----

📊 Results:

→ Successfully validated on MMLU, GSM8K, CommonsenseQA, and CoLA benchmarks

→ Accurately predicts emergence up to 4x compute in advance

→ Demonstrates practical applications in assessing pretraining data quality