"Densing Law of LLMs"

The exponential march towards more efficient language models.

This paper introduces "capability density" as a new metric for evaluating LLM quality across different scales, revealing that model density doubles every 3 months. This discovery, termed the "Densing Law," provides insights into LLM development trends and efficiency improvements.

-----

https://arxiv.org/abs/2412.04315

🤔 Original Problem:

→ Traditional LLM evaluation focuses solely on performance scaling with size, neglecting efficiency and practical deployment constraints.

→ The field lacks quantitative metrics to evaluate LLMs of different scales while considering both effectiveness and computational efficiency.

-----

⚡ Solution in this Paper:

→ The paper introduces "capability density" - defined as the ratio of effective parameter size to actual parameter size.

→ Effective parameter size is calculated using reference models and scaling functions to predict downstream performance.

→ The methodology employs a two-step estimation process: first fitting loss estimation, then performance estimation using sigmoid functions.

-----

🔍 Key Insights:

→ LLM density exhibits exponential growth, doubling approximately every 3.3 months

→ Inference costs decrease exponentially for equivalent performance levels

→ Model compression methods often result in lower density than original models

-----

📊 Results:

→ Maximum density growth rate (A ≈ 0.007) with R² ≈ 0.93

→ Inference costs reduced by 266.7x from January 2023 to present

→ Density growth accelerated by 50% after ChatGPT's release