readme: ---
  language: 
  - en
  
  tags:
  - text2text-generation
  
  widget:
  - text: "The <extra_id_0> walks in <extra_id_1> park"
    example_title: "Masked Language Modeling"
  
  datasets:
  - c4
  
  
  license: apache-2.0
  ---
  
  # Model Card for Switch Transformers Large - 128 experts
  
  ![model image](https://s3.amazonaws.com/moonup/production/uploads/1666966931908-62441d1d9fdefb55a0b7d12c.png)
  
  #  Table of Contents
  
  0. [TL;DR](#TL;DR)
  1. [Model Details](#model-details)
  2. [Usage](#usage)
  3. [Uses](#uses)
  4. [Bias, Risks, and Limitations](#bias-risks-and-limitations)
  5. [Training Details](#training-details)
  6. [Evaluation](#evaluation)
  7. [Environmental Impact](#environmental-impact)
  8. [Citation](#citation)
  9. [Model Card Authors](#model-card-authors)
  
  # TL;DR
  
  Switch Transformers is a Mixture of Experts (MoE) model trained on Masked Language Modeling (MLM) task. The model architecture is similar to the classic T5, but with the Feed Forward layers replaced by the Sparse MLP layers containing "experts" MLP. According to the [original paper](https://arxiv.org/pdf/2101.03961.pdf) the model enables faster training (scaling properties) while being better than T5 on fine-tuned tasks. 
  As mentioned in the first few lines of the abstract : 
  >  we advance the current scale of language models by pre-training up to trillion parameter models on the “Colossal Clean Crawled Corpus”, and achieve a 4x speedup over the T5-XXL model.
  
  **Disclaimer**: Content from **this** model card has been written by the Hugging Face team, and parts of it were copy pasted from the [original paper](https://arxiv.org/pdf/2101.03961.pdf).
  
  # Model Details
  
  ## Model Description
  
  
  - **Model type:** Language model
  - **Language(s) (NLP):** English
  - **License:** Apache 2.0
  - **Related Models:** [All Switch Transformers Checkpoints](https://huggingface.co/models?search=switch)
  - **Original Checkpoints:** [All Original Switch Transformers Checkpoints](https://github.com/google-research/t5x/blob/main/docs/models.md#mixture-of-experts-moe-checkpoints)
  - **Resources for more information:**
    - [Research paper](https://arxiv.org/pdf/2101.03961.pdf)
    - [GitHub Repo](https://github.com/google-research/t5x)
    - [Hugging Face Switch Transformers Docs (Similar to T5) ](https://huggingface.co/docs/transformers/model_doc/switch_transformers)
  
  # Usage
  
  Note that these checkpoints has been trained on Masked-Language Modeling (MLM) task. Therefore the checkpoints are not "ready-to-use" for downstream tasks. You may want to check `FLAN-T5` for running fine-tuned weights or fine-tune your own MoE following [this notebook](https://colab.research.google.com/drive/1aGGVHZmtKmcNBbAwa9hbu58DDpIuB5O4?usp=sharing)
  
  Find below some example scripts on how to use the model in `transformers`:
  
  ## Using the Pytorch model
  
  ### Running the model on a CPU
  
  <details>
  <summary> Click to expand </summary>
  
  ```python
  
  from transformers import AutoTokenizer, SwitchTransformersForConditionalGeneration
  
  tokenizer = AutoTokenizer.from_pretrained("google/switch-large-128")
  model = SwitchTransformersForConditionalGeneration.from_pretrained("google/switch-large-128")
  
  input_text = "A <extra_id_0> walks into a bar a orders a <extra_id_1> with <extra_id_2> pinch of <extra_id_3>."
  input_ids = tokenizer(input_text, return_tensors="pt").input_ids
  
  outputs = model.generate(input_ids)
  print(tokenizer.decode(outputs[0]))
  >>> <pad> <extra_id_0> man<extra_id_1> beer<extra_id_2> a<extra_id_3> salt<extra_id_4>.</s>
  ```
  
  </details>
  
  ### Running the model on a GPU
  
  <details>
  <summary> Click to expand </summary>
  
  ```python
  # pip install accelerate
  from transformers import AutoTokenizer, SwitchTransformersForConditionalGeneration
  
  tokenizer = AutoTokenizer.from_pretrained("google/switch-large-128")
  model = SwitchTransformersForConditionalGeneration.from_pretrained("google/switch-large-128", device_map="auto")
  
  input_text = "A <extra_id_0> walks into a bar a orders a <extra_id_1> with <extra_id_2> pinch of <extra_id_3>."
  input_ids = tokenizer(input_text, return_tensors="pt").input_ids.to(0)
  
  outputs = model.generate(input_ids)
  print(tokenizer.decode(outputs[0]))
  >>> <pad> <extra_id_0> man<extra_id_1> beer<extra_id_2> a<extra_id_3> salt<extra_id_4>.</s>
  ```
  
  </details>
  
  ### Running the model on a GPU using different precisions
  
  #### FP16
  
  <details>
  <summary> Click to expand </summary>
  
  ```python
  # pip install accelerate
  from transformers import AutoTokenizer, SwitchTransformersForConditionalGeneration
  
  tokenizer = AutoTokenizer.from_pretrained("google/switch-large-128")
  model = SwitchTransformersForConditionalGeneration.from_pretrained("google/switch-large-128", device_map="auto", torch_dtype=torch.float16)
  
  input_text = "A <extra_id_0> walks into a bar a orders a <extra_id_1> with <extra_id_2> pinch of <extra_id_3>."
  input_ids = tokenizer(input_text, return_tensors="pt").input_ids.to(0)
  
  outputs = model.generate(input_ids)
  print(tokenizer.decode(outputs[0]))
  >>> <pad> <extra_id_0> man<extra_id_1> beer<extra_id_2> a<extra_id_3> salt<e