readme: ---
  language:
  - multilingual
  - en
  - es
  - fr
  - de
  - zh
  - ru 
  - pt
  - it
  - ar
  - ja 
  - id 
  - tr 
  - nl
  - pl
  - fa
  - vi
  - sv
  - ko 
  - he
  - ro
  - no
  - hi
  - uk
  - cs
  - fi
  - hu
  - th
  - da
  - ca
  - el
  - bg
  - sr
  - ms
  - bn
  - hr
  - sl
  - az
  - sk
  - eo
  - ta
  - sh
  - lt
  - et
  - ml
  - la
  - bs
  - sq
  - arz
  - af
  - ka
  - mr
  - eu
  - tl
  - ang
  - gl
  - nn
  - ur
  - kk
  - be
  - hy
  - te
  - lv
  - mk
  - als
  - is
  - wuu
  - my
  - sco
  - mn
  - ceb
  - ast
  - cy
  - kn
  - br
  - an
  - gu
  - bar
  - uz
  - lb
  - ne
  - si
  - war
  - jv
  - ga
  - oc
  - ku
  - sw
  - nds
  - ckb
  - ia
  - yi
  - fy
  - scn
  - gan
  - tt
  - am
  license: cc-by-nc-4.0
  ---
  
  # xlm-mlm-100-1280
  
  #  Table of Contents
  
  1. [Model Details](#model-details)
  2. [Uses](#uses)
  3. [Bias, Risks, and Limitations](#bias-risks-and-limitations)
  4. [Training](#training)
  5. [Evaluation](#evaluation)
  6. [Environmental Impact](#environmental-impact)
  7. [Technical Specifications](#technical-specifications)
  8. [Citation](#citation)
  9. [Model Card Authors](#model-card-authors)
  10. [How To Get Started With the Model](#how-to-get-started-with-the-model)
  
  
  # Model Details
  
  xlm-mlm-100-1280 is the XLM model, which was proposed in [Cross-lingual Language Model Pretraining](https://arxiv.org/abs/1901.07291) by Guillaume Lample and Alexis Conneau, trained on Wikipedia text in 100 languages. The model is a transformer pretrained using a masked language modeling (MLM) objective. 
  
  ## Model Description
  
  - **Developed by:** See [associated paper](https://arxiv.org/abs/1901.07291) and [GitHub Repo](https://github.com/facebookresearch/XLM)
  - **Model type:** Language model
  - **Language(s) (NLP):** 100 languages, see [GitHub Repo](https://github.com/facebookresearch/XLM#the-17-and-100-languages) for full list.
  - **License:** CC-BY-NC-4.0
  - **Related Models:** [xlm-mlm-17-1280](https://huggingface.co/xlm-mlm-17-1280)
  - **Resources for more information:** 
    - [Associated paper](https://arxiv.org/abs/1901.07291)
    - [GitHub Repo](https://github.com/facebookresearch/XLM#the-17-and-100-languages)
    - [Hugging Face Multilingual Models for Inference docs](https://huggingface.co/docs/transformers/v4.20.1/en/multilingual#xlm-with-language-embeddings)
  
  # Uses
  
  ## Direct Use
  
  The model is a language model. The model can be used for masked language modeling. 
  
  ## Downstream Use
  
  To learn more about this task and potential downstream uses, see the Hugging Face [fill mask docs](https://huggingface.co/tasks/fill-mask) and the [Hugging Face Multilingual Models for Inference](https://huggingface.co/docs/transformers/v4.20.1/en/multilingual#xlm-with-language-embeddings) docs. Also see the [associated paper](https://arxiv.org/abs/1901.07291).
  
  ## Out-of-Scope Use
  
  The model should not be used to intentionally create hostile or alienating environments for people. 
  
  # Bias, Risks, and Limitations
  
  Significant research has explored bias and fairness issues with language models (see, e.g., [Sheng et al. (2021)](https://aclanthology.org/2021.acl-long.330.pdf) and [Bender et al. (2021)](https://dl.acm.org/doi/pdf/10.1145/3442188.3445922)).
  
  ## Recommendations
  
  Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model.
  
  # Training
  
  This model is the XLM model trained on Wikipedia text in 100 languages. The preprocessing included tokenization with byte-pair-encoding. See the [GitHub repo](https://github.com/facebookresearch/XLM#the-17-and-100-languages) and the [associated paper](https://arxiv.org/pdf/1911.02116.pdf) for further details on the training data and training procedure.
  
  [Conneau et al. (2020)](https://arxiv.org/pdf/1911.02116.pdf) report that this model has 16 layers, 1280 hidden states, 16 attention heads, and the dimension of the feed-forward layer is 1520. The vocabulary size is 200k and the total number of parameters is 570M (see Table 7).
   
  # Evaluation
  
  ## Testing Data, Factors & Metrics
  
  The model developers evaluated the model on the XNLI cross-lingual classification task (see the [XNLI data card](https://huggingface.co/datasets/xnli) for more details on XNLI) using the metric of test accuracy. See the [GitHub Repo](https://arxiv.org/pdf/1911.02116.pdf) for further details on the testing data, factors and metrics.
  
  ## Results
  
  For xlm-mlm-100-1280, the test accuracy on the XNLI cross-lingual classification task in English (en), Spanish (es), German (de), Arabic (ar), Chinese (zh) and Urdu (ur) are:
  
  |Language| en | es  | de | ar | zh | ur |
  |:------:|:--:|:---:|:--:|:--:|:--:|:--:|
  |        |83.7|76.6 |73.6|67.4|71.7|62.9|
  
  See the [GitHub repo](https://github.com/facebookresearch/XLM#ii-cross-lingual-language-model-pretraining-xlm) for further details.
  
  # Environmental Impact
  
  Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
  
  - **Hardware Type:** More information needed
  - **Hours used:** More information needed
  - **Cloud Provider:** More information needed
  - **Compute Region:** More infor