readme: ---
  language: 
  - multilingual
  - en
  - fr 
  - es 
  - de 
  - el 
  - bg 
  - ru 
  - tr 
  - ar 
  - vi 
  - th 
  - zh 
  - hi 
  - sw 
  - ur
  license: cc-by-nc-4.0
  ---
  
  # xlm-mlm-tlm-xnli15-1024
  
  #  Table of Contents
  
  1. [Model Details](#model-details)
  2. [Uses](#uses)
  3. [Bias, Risks, and Limitations](#bias-risks-and-limitations)
  4. [Training Details](#training-details)
  5. [Evaluation](#evaluation)
  6. [Environmental Impact](#environmental-impact)
  7. [Technical Specifications](#technical-specifications)
  8. [Citation](#citation)
  9. [Model Card Authors](#model-card-authors)
  10. [How To Get Started With the Model](#how-to-get-started-with-the-model)
  
  
  # Model Details
  
  The XLM model was proposed in [Cross-lingual Language Model Pretraining](https://arxiv.org/abs/1901.07291) by Guillaume Lample, Alexis Conneau. xlm-mlm-tlm-xnli15-1024 is a transformer pretrained using a masked language modeling (MLM) objective in combination with a translation language modeling (TLM) objective and then fine-tuned on the English NLI dataset. The model developers evaluated the capacity of the model to make correct predictions in all 15 XNLI languages (see the [XNLI data card](https://huggingface.co/datasets/xnli) for further information on XNLI).
  
  ## Model Description
  
  - **Developed by:** Guillaume Lample, Alexis Conneau, see [associated paper](https://arxiv.org/abs/1901.07291)
  - **Model type:** Language model
  - **Language(s) (NLP):** English; evaluated in 15 languages (see [XNLI data card](https://huggingface.co/datasets/xnli))
  - **License:** CC-BY-NC-4.0
  - **Related Models:** [XLM models](https://huggingface.co/models?sort=downloads&search=xlm)
  - **Resources for more information:** 
    - [Associated paper](https://arxiv.org/abs/1901.07291)
    - [GitHub Repo for XLM](https://github.com/facebookresearch/XLM)
    - [GitHub Repo for XNLI](https://github.com/facebookresearch/XNLI)
    - [XNLI data card](https://huggingface.co/datasets/xnli)
    - [Hugging Face Multilingual Models for Inference docs](https://huggingface.co/docs/transformers/v4.20.1/en/multilingual#xlm-with-language-embeddings)
  
  # Uses
  
  ## Direct Use
  
  The model is a language model. The model can be used for cross-lingual text classification. Though the model is fine-tuned based on English text data, the model's ability to classify sentences in 14 other languages has been evaluated (see [Evaluation](#evaluation)).
  
  ## Downstream Use
  
  This model can be used for downstream tasks related to natural language inference in different languages. For more information, see the [associated paper](https://arxiv.org/abs/1901.07291).
  
  ## Out-of-Scope Use
  
  The model should not be used to intentionally create hostile or alienating environments for people. 
  
  # Bias, Risks, and Limitations
  
  Significant research has explored bias and fairness issues with language models (see, e.g., [Sheng et al. (2021)](https://aclanthology.org/2021.acl-long.330.pdf) and [Bender et al. (2021)](https://dl.acm.org/doi/pdf/10.1145/3442188.3445922)).
  
  ## Recommendations
  
  Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model.
  
  # Training Details
  
  Training details are culled from the [associated paper](https://arxiv.org/pdf/1901.07291.pdf). See the paper for links, citations, and further details. Also see the associated [GitHub Repo](https://github.com/facebookresearch/XLM#ii-cross-lingual-language-model-pretraining-xlm) for further details.
  
  ## Training Data
  
  The model developers write: 
  
  > We use WikiExtractor2 to extract raw sentences from Wikipedia dumps and use them as mono-lingual data for the CLM and MLM objectives. For the TLM objective, we only use parallel data that involves English, similar to Conneau et al. (2018b). 
  > - Precisely, we use MultiUN (Ziemski et al., 2016) for French, Spanish, Russian, Arabic and Chinese, and the IIT Bombay corpus (Anoop et al., 2018) for Hindi. 
  > - We extract the following corpora from the OPUS 3 website Tiedemann (2012): the EUbookshop corpus for German, Greek and Bulgarian, OpenSubtitles 2018 for Turkish, Vietnamese and Thai, Tanzil for both Urdu and Swahili and GlobalVoices for Swahili. 
  > - For Chinese, Japanese and Thai we use the tokenizer of Chang et al. (2008), the Kytea4 tokenizer, and the PyThaiNLP5 tokenizer respectively. 
  > - For all other languages, we use the tokenizer provided by Moses (Koehn et al., 2007), falling back on the default English tokenizer when necessary.
  
  For fine-tuning, the developers used the English NLI dataset (see the [XNLI data card](https://huggingface.co/datasets/xnli)).
  
  ## Training Procedure
  
  ### Preprocessing
  
  The model developers write:
  
  > We use fastBPE to learn BPE codes and split words into subword units. The BPE codes are learned on the concatenation of sentences sampled from all languages, following the method presented in Section 3.1.
  
  ### Speeds, Sizes, Times
  
  The model developers write: 
  
  > We use a Transformer architecture with 1024 hidden units, 8 heads, GELU activations (Hendrycks and Gimpel, 201