Mathematical proof shows why bigger transformers get smarter: it's all about data's hidden dimensions
Hidden geometry controlling transformer model performance
https://arxiv.org/abs/2411.06646
Original Problem 🤔:
Understanding why transformer models follow power-law scaling behavior lacks rigorous mathematical explanation, especially when input data lies on low-dimensional manifolds.
-----
Solution in this Paper 🔧:
→ Developed a mathematical framework showing transformers can universally approximate Hölder continuous functions on d-dimensional manifolds using O(log(d)) depth
→ Established statistical estimation theory proving generalization error bounds exponentially dependent on intrinsic data dimension d
→ Created novel computation method for covering number of transformer network classes
→ Validated theoretical predictions through experiments on three datasets: OpenWebText, SQL portion of The Stack, and Tiny Stories
-----
Key Insights 🎯:
→ Intrinsic data dimension d is crucial for determining model performance and scaling behavior
→ Transformers need only O(log(d)) layers for function approximation, independent of accuracy
→ Scaling laws accurately reflect dataset complexity - simpler datasets show faster convergence
→ The estimated intrinsic dimension d remains stable across model architectures and sizes
-----
Results 📊:
→ Close agreement (±0.02) between predicted and observed data scaling exponents
→ Achieved O(log(d)) depth efficiency, outperforming traditional ReLU networks
→ Successfully validated on three datasets with varying complexities, showing faster convergence for simpler datasets
