Exponentially Faster Language Modelling
Language models only really need to use an exponential fraction of their
neurons for individual inferences. As proof, we present FastBERT, a BERT
variant that uses 0.3\% of its neurons during inference while performing on par
with similar BERT models. FastBERT selectively engages just 12 out of 4095
neurons for each layer inference. This is achieved by replacing feedforward
networks with fast feedforward networks (FFFs). While no truly efficient
implementation currently exists to unlock the full acceleration potential of
conditional neural execution, we provide high-level CPU code achieving 78x
speedup over the optimized baseline feedforward implementation, and a PyTorch
implementation delivering 40x speedup over the equivalent batched feedforward
inference. We publish our training code, benchmarking setup, and model weights.
655c1d9b58eaf1464320273a
Peter Belcak
admin_assigned
2023-11-21T10:56:55.086Z
63e8cccddd2c4effdd6283cf
/avatars/b289d4dda0aafd60af3f14e19837b69c.svg
Peter Belcak
user
pbelcak
655c1d9b58eaf1464320273b
Roger Wattenhofer
admin_assigned
2023-11-21T10:57:01.606Z
64afd0c109d6573c1d89dbf7
/avatars/ed6bd7133517590f46847a0930d7a049.svg
Roger Wattenhofer
user
Rogrrr
Exponentially Faster Language Modelling
Language models only really need to use an exponential fraction of their
neurons for individual inferences. As proof, we present FastBERT, a BERT
variant that uses 0.3\% of its neurons during inference while performing on par
with similar BERT models. FastBERT selectively engages just 12 out of 4095
neurons for each layer inference. This is achieved by replacing feedforward
networks with fast feedforward networks (FFFs). While no truly efficient
implementation currently exists to unlock the full acceleration potential of
conditional neural execution, we provide high-level CPU code achieving 78x
speedup over the optimized baseline feedforward implementation, and a PyTorch
implementation delivering 40x speedup over the equivalent batched feedforward
inference. We publish our training code, benchmarking setup, and model weights.
FastBERT achieves significant inference speedup over BERT by selectively engaging a small fraction of neurons using fast feedforward networks.
FastBERT BERT feedforward networks fast feedforward networks (FFFs) conditional neural execution inference speedup