title: Adaptive Frequency Filters As Efficient Global Token Mixers
summary: Recent vision transformers, large-kernel CNNs and MLPs have attained
  remarkable successes in broad vision tasks thanks to their effective
  information fusion in the global scope. However, their efficient deployments,
  especially on mobile devices, still suffer from noteworthy challenges due to
  the heavy computational costs of self-attention mechanisms, large kernels, or
  fully connected layers. In this work, we apply conventional convolution theorem
  to deep learning for addressing this and reveal that adaptive frequency filters
  can serve as efficient global token mixers. With this insight, we propose
  Adaptive Frequency Filtering (AFF) token mixer. This neural operator transfers
  a latent representation to the frequency domain via a Fourier transform and
  performs semantic-adaptive frequency filtering via an elementwise
  multiplication, which mathematically equals to a token mixing operation in the
  original latent space with a dynamic convolution kernel as large as the spatial
  resolution of this latent representation. We take AFF token mixers as primary
  neural operators to build a lightweight neural network, dubbed AFFNet.
  Extensive experiments demonstrate the effectiveness of our proposed AFF token
  mixer and show that AFFNet achieve superior accuracy and efficiency trade-offs
  compared to other lightweight network designs on broad visual tasks, including
  visual recognition and dense prediction tasks.
paper_authors._id: 64c244d8db947599b4ba220b
paper_authors.name: Zhipeng Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-27T11:42:16.289Z
paper_authors._id: 635629096b58fa7cc87476ee
paper_authors.avatarUrl: /avatars/3d5291741cdcd26cb7cc737f76d1e3ad.svg
paper_authors.fullname: Zhipeng Huang
paper_authors.type: user
paper_authors.user: woli
paper_authors._id: 64c244d8db947599b4ba220c
paper_authors.name: Zhizheng Zhang
paper_authors._id: 64c244d8db947599b4ba220d
paper_authors.name: Cuiling Lan
paper_authors._id: 64c244d8db947599b4ba220e
paper_authors.name: Zheng-Jun Zha
paper_authors._id: 64c244d8db947599b4ba220f
paper_authors.name: Yan Lu
paper_authors._id: 64c244d8db947599b4ba2210
paper_authors.name: Baining Guo
paper_title: Adaptive Frequency Filters As Efficient Global Token Mixers
paper_summary: Recent vision transformers, large-kernel CNNs and MLPs have attained
  remarkable successes in broad vision tasks thanks to their effective
  information fusion in the global scope. However, their efficient deployments,
  especially on mobile devices, still suffer from noteworthy challenges due to
  the heavy computational costs of self-attention mechanisms, large kernels, or
  fully connected layers. In this work, we apply conventional convolution theorem
  to deep learning for addressing this and reveal that adaptive frequency filters
  can serve as efficient global token mixers. With this insight, we propose
  Adaptive Frequency Filtering (AFF) token mixer. This neural operator transfers
  a latent representation to the frequency domain via a Fourier transform and
  performs semantic-adaptive frequency filtering via an elementwise
  multiplication, which mathematically equals to a token mixing operation in the
  original latent space with a dynamic convolution kernel as large as the spatial
  resolution of this latent representation. We take AFF token mixers as primary
  neural operators to build a lightweight neural network, dubbed AFFNet.
  Extensive experiments demonstrate the effectiveness of our proposed AFF token
  mixer and show that AFFNet achieve superior accuracy and efficiency trade-offs
  compared to other lightweight network designs on broad visual tasks, including
  visual recognition and dense prediction tasks.
paper_ai_keywords: vision transformers large-kernel CNNs MLPs self-attention mechanisms adaptive frequency filters Adaptive Frequency Filtering (AFF) token mixer Fourier transform semantic-adaptive frequency filtering dynamic convolution kernel AFFNet AFF token mixer visual recognition dense prediction tasks