readme: language: en
  license: bsd
  datasets:
  - bookcorpus
  - wikipedia
  ---
  
  # SqueezeBERT pretrained model
  
  This model, `squeezebert-uncased`, is a pretrained model for the English language using a masked language modeling (MLM) and Sentence Order Prediction (SOP) objective.
  SqueezeBERT was introduced in [this paper](https://arxiv.org/abs/2006.11316). This model is case-insensitive. The model architecture is similar to BERT-base, but with the pointwise fully-connected layers replaced with [grouped convolutions](https://blog.yani.io/filter-group-tutorial/).
  The authors found that SqueezeBERT is 4.3x faster than `bert-base-uncased` on a Google Pixel 3 smartphone.
  
  
  ## Pretraining
  
  ### Pretraining data
  - [BookCorpus](https://yknzhu.wixsite.com/mbweb), a dataset consisting of thousands of unpublished books
  - [English Wikipedia](https://en.wikipedia.org/wiki/English_Wikipedia)
  
  ### Pretraining procedure
  The model is pretrained using the Masked Language Model (MLM) and Sentence Order Prediction (SOP) tasks.
  (Author's note: If you decide to pretrain your own model, and you prefer to train with MLM only, that should work too.)
  
  From the SqueezeBERT paper:
  > We pretrain SqueezeBERT from scratch (without distillation) using the [LAMB](https://arxiv.org/abs/1904.00962) optimizer, and we employ the hyperparameters recommended by the LAMB authors: a global batch size of 8192, a learning rate of 2.5e-3, and a warmup proportion of 0.28. Following the LAMB paper's recommendations, we pretrain for 56k steps with a maximum sequence length of 128 and then for 6k steps with a maximum sequence length of 512.
  
  ## Finetuning
  
  The SqueezeBERT paper results from 2 approaches to finetuning the model:
  - "finetuning without bells and whistles" -- after pretraining the SqueezeBERT model, finetune it on each GLUE task
  - "finetuning with bells and whistles" -- after pretraining the SqueezeBERT model, finetune it on a MNLI with distillation from a teacher model. Then, use the MNLI-finetuned SqueezeBERT model as a student model to finetune on each of the other GLUE tasks (e.g. RTE, MRPC, …) with distillation from a task-specific teacher model.
  
  A detailed discussion of the hyperparameters used for finetuning is provided in the appendix of the [SqueezeBERT paper](https://arxiv.org/abs/2006.11316).
  Note that finetuning SqueezeBERT with distillation is not yet implemented in this repo. If the author (Forrest Iandola - forrest.dnn@gmail.com) gets enough encouragement from the user community, he will add example code to Transformers for finetuning SqueezeBERT with distillation.
  
  This model, `squeezebert/squeezebert-uncased`, has been pretrained but not finetuned. For most text classification tasks, we recommend using squeezebert-mnli-headless as a starting point.
  
  ### How to finetune
  To try finetuning SqueezeBERT on the [MRPC](https://www.microsoft.com/en-us/download/details.aspx?id=52398) text classification task, you can run the following command:
  ```
  ./utils/download_glue_data.py
  
  python examples/text-classification/run_glue.py \
      --model_name_or_path squeezebert-base-headless \
      --task_name mrpc \
      --data_dir ./glue_data/MRPC \
      --output_dir ./models/squeezebert_mrpc \
      --overwrite_output_dir \
      --do_train \
      --do_eval \
      --num_train_epochs 10 \
      --learning_rate 3e-05 \
      --per_device_train_batch_size 16 \
      --save_steps 20000
  
  ```
  
  ## BibTeX entry and citation info
  ```
  @article{2020_SqueezeBERT,
       author = {Forrest N. Iandola and Albert E. Shaw and Ravi Krishna and Kurt W. Keutzer},
       title = {{SqueezeBERT}: What can computer vision teach NLP about efficient neural networks?},
       journal = {arXiv:2006.11316},
       year = {2020}
  }
  ```
  