title: Approximating Two-Layer Feedforward Networks for Efficient Transformers
summary: How to reduce compute and memory requirements of neural networks (NNs)
  without sacrificing performance? Many recent works use sparse Mixtures of
  Experts (MoEs) to build resource-efficient large language models (LMs). Here we
  introduce several novel perspectives on MoEs, presenting a general framework
  that unifies various methods to approximate two-layer NNs (e.g., feedforward
  blocks of Transformers), including product-key memories (PKMs). Leveraging
  insights from this framework, we propose methods to improve both MoEs and PKMs.
  Unlike prior work that compares MoEs with dense baselines under the
  compute-equal condition, our evaluation condition is parameter-equal, which is
  crucial to properly evaluate LMs. We show that our MoEs are competitive with
  the dense Transformer-XL on both the WikiText-103 and enwiki8 datasets at two
  different scales, while being much more resource efficient. This demonstrates
  that MoEs are relevant not only to extremely large LMs but also to any-scale
  resource-efficient LMs. Our code is public.
paper_authors._id: 652f4e1dfd5e3a357c18d421
paper_authors.name: Róbert Csordás
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-18T13:42:57.608Z
paper_authors._id: 64e5d1bef450dca5ca0d05ed
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64e5d1bef450dca5ca0d05ed/HQHb6jM-3mmfCOzR-4eeW.jpeg
paper_authors.fullname: Robert Csordas
paper_authors.type: user
paper_authors.user: robertcsordas
paper_authors._id: 652f4e1dfd5e3a357c18d422
paper_authors.name: Kazuki Irie
paper_authors._id: 652f4e1dfd5e3a357c18d423
paper_authors.name: Jürgen Schmidhuber
paper_title: Approximating Two-Layer Feedforward Networks for Efficient Transformers
paper_summary: How to reduce compute and memory requirements of neural networks (NNs)
  without sacrificing performance? Many recent works use sparse Mixtures of
  Experts (MoEs) to build resource-efficient large language models (LMs). Here we
  introduce several novel perspectives on MoEs, presenting a general framework
  that unifies various methods to approximate two-layer NNs (e.g., feedforward
  blocks of Transformers), including product-key memories (PKMs). Leveraging
  insights from this framework, we propose methods to improve both MoEs and PKMs.
  Unlike prior work that compares MoEs with dense baselines under the
  compute-equal condition, our evaluation condition is parameter-equal, which is
  crucial to properly evaluate LMs. We show that our MoEs are competitive with
  the dense Transformer-XL on both the WikiText-103 and enwiki8 datasets at two
  different scales, while being much more resource efficient. This demonstrates
  that MoEs are relevant not only to extremely large LMs but also to any-scale
  resource-efficient LMs. Our code is public.
paper_ai_keywords: Sparse Mixtures of Experts (MoEs) large language models (LMs) two-layer NNs feedforward blocks Transformers product-key memories (PKMs) parameter-equal condition Transformer-XL WikiText-103 enwiki8