title: Exponentially Faster Language Modelling
summary: Language models only really need to use an exponential fraction of their
  neurons for individual inferences. As proof, we present FastBERT, a BERT
  variant that uses 0.3\% of its neurons during inference while performing on par
  with similar BERT models. FastBERT selectively engages just 12 out of 4095
  neurons for each layer inference. This is achieved by replacing feedforward
  networks with fast feedforward networks (FFFs). While no truly efficient
  implementation currently exists to unlock the full acceleration potential of
  conditional neural execution, we provide high-level CPU code achieving 78x
  speedup over the optimized baseline feedforward implementation, and a PyTorch
  implementation delivering 40x speedup over the equivalent batched feedforward
  inference. We publish our training code, benchmarking setup, and model weights.
paper_authors._id: 655c1d9b58eaf1464320273a
paper_authors.name: Peter Belcak
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-21T10:56:55.086Z
paper_authors._id: 63e8cccddd2c4effdd6283cf
paper_authors.avatarUrl: /avatars/b289d4dda0aafd60af3f14e19837b69c.svg
paper_authors.fullname: Peter Belcak
paper_authors.type: user
paper_authors.user: pbelcak
paper_authors._id: 655c1d9b58eaf1464320273b
paper_authors.name: Roger Wattenhofer
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-21T10:57:01.606Z
paper_authors._id: 64afd0c109d6573c1d89dbf7
paper_authors.avatarUrl: /avatars/ed6bd7133517590f46847a0930d7a049.svg
paper_authors.fullname: Roger Wattenhofer
paper_authors.type: user
paper_authors.user: Rogrrr
paper_title: Exponentially Faster Language Modelling
paper_summary: Language models only really need to use an exponential fraction of their
  neurons for individual inferences. As proof, we present FastBERT, a BERT
  variant that uses 0.3\% of its neurons during inference while performing on par
  with similar BERT models. FastBERT selectively engages just 12 out of 4095
  neurons for each layer inference. This is achieved by replacing feedforward
  networks with fast feedforward networks (FFFs). While no truly efficient
  implementation currently exists to unlock the full acceleration potential of
  conditional neural execution, we provide high-level CPU code achieving 78x
  speedup over the optimized baseline feedforward implementation, and a PyTorch
  implementation delivering 40x speedup over the equivalent batched feedforward
  inference. We publish our training code, benchmarking setup, and model weights.
paper_ai_summary: FastBERT achieves significant inference speedup over BERT by selectively engaging a small fraction of neurons using fast feedforward networks.
paper_ai_keywords: FastBERT BERT feedforward networks fast feedforward networks (FFFs) conditional neural execution inference speedup