readme: ---
  datasets:
  - olm/olm-december-2022-tokenized-512
  language:
  - en
  pipeline_tag: fill-mask
  license: apache-2.0
  ---
  
  
  # Tiny BERT December 2022
  
  This is a more up-to-date version of the [original tiny BERT](https://huggingface.co/google/bert_uncased_L-2_H-128_A-2) referenced in [Well-Read Students Learn Better: On the Importance of Pre-training Compact Models](https://arxiv.org/abs/1908.08962) (English only, uncased, trained with WordPiece masking).
  In addition to being more up-to-date, it is more CPU friendly than its base version.
  
  We think it is fair to directly compare our model to the original tiny BERT because our model was trained with about the same level of compute as the original tiny BERT.
  Our model was trained on a cleaned December 2022 snapshot of Common Crawl and Wikipedia.
  
  This model was created as part of the OLM project, which has the goal of continuously training and releasing models that are up-to-date and comparable in standard language model performance to their static counterparts.
  This is important because we want our models to know about events like COVID or 
  a presidential election right after they happen.
  
  ## Intended uses
  
  You can use the raw model for masked language modeling, but it's mostly intended to
  be fine-tuned on a downstream task, such as sequence classification, token classification or question answering.
  
  ## Dataset
  
  The model and tokenizer were trained with this [December 2022 cleaned Common Crawl dataset](https://huggingface.co/datasets/olm/olm-CC-MAIN-2022-49-sampling-ratio-olm-0.15114822547) plus this [December 2022 cleaned Wikipedia dataset](https://huggingface.co/datasets/olm/olm-wikipedia-20221220).\
  The tokenized version of these concatenated datasets is [here](https://huggingface.co/datasets/olm/olm-december-2022-tokenized-512).\
  The datasets were created with this [repo](https://github.com/huggingface/olm-datasets).
  
  ## Training
  
  The model was trained according to the OLM BERT/RoBERTa instructions at this [repo](https://github.com/huggingface/olm-training).