FAN: Fourier Analysis Networks

Bold claim in this paper. Says FAN: Fourier Analysis Network can replace MLP layers in various models

• Language modeling: Up to 14.65% lower loss, 8.50% higher accuracy vs standard Transformer

• FAN leverages Fourier Analysis to model periodicity, demonstrating improved generalization and efficiency.

📚 https://arxiv.org/pdf/2410.02675

Original Problem 🔍:

Existing neural networks struggle to model and reason about periodicity, tending to memorize periodic data rather than understand underlying principles.

-----

Solution in this Paper 💡:

• Proposes FAN: Fourier Analysis Network

• Incorporates Fourier Series into network architecture

• FAN layer: ϕ(x) = [cos(W_p x) || sin(W_p x) || σ(B_p̄ + W_p̄ x)]

-----

Key Insights from this Paper 💡:

• FAN outperforms baselines in modeling basic and complex periodic functions

• Demonstrates superior performance on real-world tasks

• Reduces parameters and FLOPs compared to MLP

• Enhances generalization in cross-domain tasks

-----

Results 📊:

• Periodicity modeling: FAN significantly outperforms MLP, KAN, Transformer

• Symbolic formula representation: FAN surpasses baselines as parameter count increases

• Time series forecasting: Transformer with FAN improves MSE by 14.3-15.0%, MAE by 7.6-7.9%