Bold claim in this paper. Says FAN: Fourier Analysis Network can replace MLP layers in various models
β’ Language modeling: Up to 14.65% lower loss, 8.50% higher accuracy vs standard Transformer
β’ FAN leverages Fourier Analysis to model periodicity, demonstrating improved generalization and efficiency.
π https://arxiv.org/pdf/2410.02675
Original Problem π:
Existing neural networks struggle to model and reason about periodicity, tending to memorize periodic data rather than understand underlying principles.
-----
Solution in this Paper π‘:
β’ Proposes FAN: Fourier Analysis Network
β’ Incorporates Fourier Series into network architecture
β’ FAN layer: Ο(x) = [cos(W_p x) || sin(W_p x) || Ο(B_pΜ + W_pΜ x)]
-----
Key Insights from this Paper π‘:
β’ FAN outperforms baselines in modeling basic and complex periodic functions
β’ Demonstrates superior performance on real-world tasks
β’ Reduces parameters and FLOPs compared to MLP
β’ Enhances generalization in cross-domain tasks
-----
Results π:
β’ Periodicity modeling: FAN significantly outperforms MLP, KAN, Transformer
β’ Symbolic formula representation: FAN surpasses baselines as parameter count increases
β’ Time series forecasting: Transformer with FAN improves MSE by 14.3-15.0%, MAE by 7.6-7.9%
