SqueezeBERT, which runs 4.3x faster than BERT-base on the Pixel 3 while achieving competitive accuracy on the GLUE test
set. The SqueezeBERT code will be released.
Tips:
SqueezeBERT is a model with absolute position embeddings so it's usually advised to pad the inputs on the right
  rather than the left.
SqueezeBERT is similar to BERT and therefore relies on the masked language modeling (MLM) objective. It is therefore
  efficient at predicting masked tokens and at NLU in general, but is not optimal for text generation. Models trained
  with a causal language modeling (CLM) objective are better in that regard.
For best results when finetuning on sequence classification tasks, it is recommended to start with the
  squeezebert/squeezebert-mnli-headless checkpoint.
This model was contributed by forresti.
