"Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models"

Sparse Mixture-of-Experts models are not just smaller; they are smarter when FLOPs are fixed.

Optimal sparsity unlocks superior LLM performance within a given computational budget.

The paper addresses efficient scaling of Mixture-of-Experts Large Language Models by investigating the interplay between model parameters and FLOPs, aiming to find optimal sparsity configurations for training. It reveals that for a fixed FLOP budget, sparse Mixture-of-Experts models outperform dense models and identifies scaling laws that guide efficient model design.

-----

Paper - https://arxiv.org/abs/2501.12370

Original Problem 🧐:

→ Efficient scaling strategies are crucial to reduce computational costs.

→ Determining the optimal balance between model size and computation remains a challenge.

-----

Solution in this Paper 💡:

→ This paper investigates scaling laws for Mixture-of-Experts LLMs, focusing on parameter and FLOP trade-offs.

→ It explores the interplay between Noisy Sparsity and Data Sparsity in Mixture-of-Experts models.

→ The paper empirically derives scaling laws that dictate the optimal sparsity level for a given FLOP budget.

→ It proposes that for a fixed computational budget, sparse Mixture-of-Experts models achieve better performance than dense models.

→ The study varies model size, number of experts, and sparsity levels to analyze their impact on performance under controlled FLOPs.

-----

Key Insights from this Paper 🤔:

→ Sparsity is key to efficient scaling of Mixture-of-Experts LLMs under fixed FLOP budgets.

→ Noisy Sparsity and Data Sparsity exhibit different scaling behaviors and impacts.

→ Optimal sparsity levels exist and are dependent on the computational budget.

→ Mixture-of-Experts models with carefully tuned sparsity outperform their dense counterparts at the same FLOP cost.

→ Scaling laws can guide the design of computationally efficient and performant Mixture-of-Experts LLMs.

-----

Results 📊:

→ Sparse Mixture-of-Experts models achieve comparable or better performance than dense models with significantly fewer parameters, under similar FLOP regimes.

→ The derived scaling laws accurately predict performance trends across different sparsity levels and model sizes.

→ Empirical results validate the theoretical insights regarding the parameter-FLOP trade-off in sparse Mixture-of-Experts LLMs.