readme: 
  # Model Card for COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining
  
   
  # Model Details
   
  ## Model Description
   
  This model card contains the COCO-LM model (**base++** version) pretrained models on GLUE and SQuAD 2.0 benchmarks.
   
  - **Developed by:** Microsoft
  - **Shared by [Optional]:** HuggingFace
  - **Model type:** Language model
  - **Language(s) (NLP):** en
  - **License:** MIT
  - **Related Models:** More information needed
    - **Parent Model:** More information needed
  - **Resources for more information:** 
   - [GitHub Repo](https://github.com/microsoft/COCO-LM)
   - [Associated Paper](https://arxiv.org/abs/2102.08473)
   
  # Uses
   
   
  ## Direct Use
   
  Correcting and Contrasting Text Sequences for Language Model Pretraining
   
  ## Downstream Use [Optional]
   
  More information needed.
   
  ## Out-of-Scope Use
   
   
  The model should not be used to intentionally create hostile or alienating environments for people. 
   
  # Bias, Risks, and Limitations
   
   
  Significant research has explored bias and fairness issues with language models (see, e.g., [Sheng et al. (2021)](https://aclanthology.org/2021.acl-long.330.pdf) and [Bender et al. (2021)](https://dl.acm.org/doi/pdf/10.1145/3442188.3445922)). Predictions generated by the model may include disturbing and harmful stereotypes across protected classes; identity characteristics; and sensitive, social, and occupational groups.
   
   
  ## Recommendations
   
   
  Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recomendations.
   
   
  # Training Details
   
  ## Training Data
   
   
  See the associated [dataset card]({0}) for further details
   
  ## Training Procedure
   
  ### Preprocessing
   
  The model devloeprs note in the [associated paper](https://arxiv.org/abs/2102.08473):
   
  >We employ three standard settings, base, base++, and large++. Base is the BERTBase training configuration: Pretraining on Wikipedia and BookCorpus (16 GB of texts) for 256 million samples on 512 token sequences (125K batches with 2048 batch size). We use the same corpus and 32, 768 uncased BPE vocabulary as with TUPE. Base++ trains the base size model with larger corpora and/or more training steps. Following recent research, we add in OpenWebText, CC-News , and STORIES, to a total of 160 GB texts, and train for 4 billion (with 2048 batch size) samples. We follow the prepossessing of UniLMV2 and use 64, 000 cased BPE vocabulary. Large++ uses the same training corpora as base++ and pretrains for 4 billion samples (2048 batch size). Its Transformer configuration is the same with BERTLarge.
   
  ### Speeds, Sizes, Times
   
  More information needed
   
  # Evaluation
   
   
   
  ## Testing Data, Factors & Metrics
   
  ### Testing Data
   
  GLUE 
  SQuAD 2.0
   
  ### Factors
   
  All results are single-task, single-model fine-tuning.
   
  ### Metrics
   
  More information needed
   
  ## Results 
   
  ### GLUE Fine-Tuning Results
   
  The [General Language Understanding Evaluation (GLUE)](https://gluebenchmark.com/) benchmark is a collection of sentence- or sentence-pair language understanding tasks for evaluating and analyzing natural language understanding systems. 
   
  GLUE dev set results of COCO-LM base++ and large++ models are as follows (median of 5 different random seeds):
   
  | Model | MNLI-m/mm | QQP | QNLI | SST-2 | CoLA | RTE | MRPC | STS-B | AVG |
  | ------ | ------ | ------ | ------ | ------ | ------ | ------ | ------ | ------ | ------ |
  | COCO-LM base++ | 90.2/90.0 | 92.2 | 94.2 | 94.6 | 67.3 | 87.4 | 91.2 | 91.8 | 88.6 |
  | COCO-LM large++ | 91.4/91.6 | 92.8 | 95.7 | 96.9 | 73.9 | 91.0 | 92.2 | 92.7 | 90.8 |
   
  GLUE test set results of COCO-LM base++ and large++ models are as follows (no ensemble, task-specific tricks, etc.):
   
  | Model | MNLI-m/mm | QQP | QNLI | SST-2 | CoLA | RTE | MRPC | STS-B | AVG |
  | ------ | ------ | ------ | ------ | ------ | ------ | ------ | ------ | ------ | ------ |
  | COCO-LM base++ | 89.8/89.3 | 89.8 | 94.2 | 95.6 | 68.6 | 82.3 | 88.5 | 90.3 | 87.4 |
  | COCO-LM large++ | 91.6/91.1 | 90.5 | 95.8 | 96.7 | 70.5 | 89.2 | 88.4 | 91.8 | 89.3 |
   
  ### SQuAD 2.0 Fine-Tuning Results
   
  [Stanford Question Answering Dataset (SQuAD)](https://rajpurkar.github.io/SQuAD-explorer/) is a reading comprehension dataset, consisting of questions posed by crowdworkers on a set of Wikipedia articles, where the answer to every question is a segment of text, or span, from the corresponding reading passage, or the question might be unanswerable. 
   
  SQuAD 2.0 dev set results of COCO-LM base++ and large++ models are as follows (median of 5 different random seeds):
   
  | Model | EM | F1 |
  | ------ | ------ | ------ |
  | COCO-LM base++ | 85.4 | 88.1 |
  | COCO-LM large++ | 88.2 | 91.0 |
   
   
   
   
  # Model Examination
   
  The model delevopers note in the [associated paper](https://arxiv.org/abs/2102.08473):
   
  >Architecture. Removing relative position encoding (Rel-Pos) leads to better numbers on some tasks but significantly hurts MNLI. Using a shallow auxiliary network 