FAN: Fourier Analysis Networks
Despite the remarkable success achieved by neural networks, particularly
those represented by MLP and Transformer, we reveal that they exhibit potential
flaws in the modeling and reasoning of periodicity, i.e., they tend to memorize
the periodic data rather than genuinely understanding the underlying principles
of periodicity. However, periodicity is a crucial trait in various forms of
reasoning and generalization, underpinning predictability across natural and
engineered systems through recurring patterns in observations. In this paper,
we propose FAN, a novel network architecture based on Fourier Analysis, which
empowers the ability to efficiently model and reason about periodic phenomena.
By introducing Fourier Series, the periodicity is naturally integrated into the
structure and computational processes of the neural network, thus achieving a
more accurate expression and prediction of periodic patterns. As a promising
substitute to multi-layer perceptron (MLP), FAN can seamlessly replace MLP in
various models with fewer parameters and FLOPs. Through extensive experiments,
we demonstrate the effectiveness of FAN in modeling and reasoning about
periodic functions, and the superiority and generalizability of FAN across a
range of real-world tasks, including symbolic formula representation, time
series forecasting, and language modeling.
6700c883876cd40f132c439a
Yihong Dong
admin_assigned
2024-10-08T09:23:39.018Z
6468e631b2321e47d3277ec9
/avatars/3acd3868232067289fad69db6c32cf87.svg
Yihong Dong
user
dongyh
6700c883876cd40f132c439b
Ge Li
claimed_verified
2024-10-10T08:02:09.324Z
6530b96ef68c6e013d08b413
/avatars/ecc2874e612d0aed357d072427faf21a.svg
Ge Li
user
ligechina
6700c883876cd40f132c439c
Yongding Tao
claimed_verified
2024-12-09T14:48:39.714Z
664b3aebfe822b08e62357f0
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/2N3Mu7q3hV_WxELmTbrld.jpeg
Yongding Tao
user
YongdingTao
6700c883876cd40f132c439d
Xue Jiang
6700c883876cd40f132c439e
Kechi Zhang
admin_assigned
2024-10-08T09:24:24.072Z
62611f7c8fd541272cfa6a2d
/avatars/c7b3cf80a9a1c78802faa97b569aba4c.svg
Kechi Zhang
user
zkcpku
6700c883876cd40f132c439f
Jia Li
6700c883876cd40f132c43a0
Jing Su
6700c883876cd40f132c43a1
Jun Zhang
6700c883876cd40f132c43a2
Jingjing Xu
admin_assigned
2024-10-08T09:25:02.987Z
64dc74c83ae9693d991f456c
/avatars/395710d328a824a8a84f4b78babe5809.svg
Jingjing Xu
user
Jingjingxu
FAN: Fourier Analysis Networks
Despite the remarkable success achieved by neural networks, particularly
those represented by MLP and Transformer, we reveal that they exhibit potential
flaws in the modeling and reasoning of periodicity, i.e., they tend to memorize
the periodic data rather than genuinely understanding the underlying principles
of periodicity. However, periodicity is a crucial trait in various forms of
reasoning and generalization, underpinning predictability across natural and
engineered systems through recurring patterns in observations. In this paper,
we propose FAN, a novel network architecture based on Fourier Analysis, which
empowers the ability to efficiently model and reason about periodic phenomena.
By introducing Fourier Series, the periodicity is naturally integrated into the
structure and computational processes of the neural network, thus achieving a
more accurate expression and prediction of periodic patterns. As a promising
substitute to multi-layer perceptron (MLP), FAN can seamlessly replace MLP in
various models with fewer parameters and FLOPs. Through extensive experiments,
we demonstrate the effectiveness of FAN in modeling and reasoning about
periodic functions, and the superiority and generalizability of FAN across a
range of real-world tasks, including symbolic formula representation, time
series forecasting, and language modeling.
A new Fourier-based network architecture, FAN, efficiently models periodic phenomena with fewer parameters and demonstrates superior performance across various tasks.
FAN Fourier Analysis Fourier Series periodic phenomena symbolic formula representation time series forecasting language modeling