readme: ---
  language: 
  - en
  - fr
  - ro
  - de
  - multilingual
  license: apache-2.0
  tags:
  - summarization
  - translation
  datasets:
  - c4
  ---
  
  # Model Card for T5-3B
  
  ![model image](https://camo.githubusercontent.com/623b4dea0b653f2ad3f36c71ebfe749a677ac0a1/68747470733a2f2f6d69726f2e6d656469756d2e636f6d2f6d61782f343030362f312a44304a31674e51663876727255704b657944387750412e706e67)
  
  #  Table of Contents
  
  1. [Model Details](#model-details)
  2. [Uses](#uses)
  3. [Bias, Risks, and Limitations](#bias-risks-and-limitations)
  4. [Training Details](#training-details)
  5. [Evaluation](#evaluation)
  6. [Environmental Impact](#environmental-impact)
  7. [Citation](#citation)
  8. [Model Card Authors](#model-card-authors)
  9. [How To Get Started With the Model](#how-to-get-started-with-the-model)
  
  # Model Details
  
  ## Model Description
  
  The developers of the Text-To-Text Transfer Transformer (T5) [write](https://ai.googleblog.com/2020/02/exploring-transfer-learning-with-t5.html): 
  
  > With T5, we propose reframing all NLP tasks into a unified text-to-text-format where the input and output are always text strings, in contrast to BERT-style models that can only output either a class label or a span of the input. Our text-to-text framework allows us to use the same model, loss function, and hyperparameters on any NLP task.
  
  T5-3B is the checkpoint with 3 billion parameters. 
  
  - **Developed by:** Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J. Liu. See [associated paper](https://jmlr.org/papers/volume21/20-074/20-074.pdf) and [GitHub repo](https://github.com/google-research/text-to-text-transfer-transformer#released-model-checkpoints)
  - **Model type:** Language model
  - **Language(s) (NLP):** English, French, Romanian, German
  - **License:** Apache 2.0
  - **Related Models:** [All T5 Checkpoints](https://huggingface.co/models?search=t5)
  - **Resources for more information:**
    - [Research paper](https://jmlr.org/papers/volume21/20-074/20-074.pdf)
    - [Google's T5 Blog Post](https://ai.googleblog.com/2020/02/exploring-transfer-learning-with-t5.html) 
    - [GitHub Repo](https://github.com/google-research/text-to-text-transfer-transformer)
    - [Hugging Face T5 Docs](https://huggingface.co/docs/transformers/model_doc/t5)
    
  # Uses
  
  ## Direct Use and Downstream Use
  
  The developers write in a [blog post](https://ai.googleblog.com/2020/02/exploring-transfer-learning-with-t5.html) that the model: 
  
  > Our text-to-text framework allows us to use the same model, loss function, and hyperparameters on any NLP task, including machine translation, document summarization, question answering, and classification tasks (e.g., sentiment analysis). We can even apply T5 to regression tasks by training it to predict the string representation of a number instead of the number itself.
  
  See the [blog post](https://ai.googleblog.com/2020/02/exploring-transfer-learning-with-t5.html) and [research paper](https://jmlr.org/papers/volume21/20-074/20-074.pdf) for further details.
  
  ## Out-of-Scope Use
  
  More information needed.
  
  # Bias, Risks, and Limitations
  
  More information needed.
  
  ## Recommendations
  
  More information needed.
  
  # Training Details
  
  ## Training Data
  
  The model is pre-trained on the [Colossal Clean Crawled Corpus (C4)](https://www.tensorflow.org/datasets/catalog/c4), which was developed and released in the context of the same [research paper](https://jmlr.org/papers/volume21/20-074/20-074.pdf) as T5.
  
  The model was pre-trained on a on a **multi-task mixture of unsupervised (1.) and supervised tasks (2.)**.
  Thereby, the following datasets were being used for (1.) and (2.):
  
  1. **Datasets used for Unsupervised denoising objective**:
  
  - [C4](https://huggingface.co/datasets/c4)
  - [Wiki-DPR](https://huggingface.co/datasets/wiki_dpr)
  
  
  2. **Datasets used for Supervised text-to-text language modeling objective**
  
  - Sentence acceptability judgment
    - CoLA [Warstadt et al., 2018](https://arxiv.org/abs/1805.12471)
  - Sentiment analysis 
    - SST-2 [Socher et al., 2013](https://nlp.stanford.edu/~socherr/EMNLP2013_RNTN.pdf)
  - Paraphrasing/sentence similarity
    - MRPC [Dolan and Brockett, 2005](https://aclanthology.org/I05-5002)
    - STS-B [Ceret al., 2017](https://arxiv.org/abs/1708.00055)
    - QQP [Iyer et al., 2017](https://quoradata.quora.com/First-Quora-Dataset-Release-Question-Pairs)
  - Natural language inference
    - MNLI [Williams et al., 2017](https://arxiv.org/abs/1704.05426)
    - QNLI [Rajpurkar et al.,2016](https://arxiv.org/abs/1606.05250)
    - RTE [Dagan et al., 2005](https://link.springer.com/chapter/10.1007/11736790_9) 
    - CB [De Marneff et al., 2019](https://semanticsarchive.net/Archive/Tg3ZGI2M/Marneffe.pdf)
  - Sentence completion
    - COPA [Roemmele et al., 2011](https://www.researchgate.net/publication/221251392_Choice_of_Plausible_Alternatives_An_Evaluation_of_Commonsense_Causal_Reasoning)
  - Word sense disambiguation
    - WIC [Pilehvar and Camacho-Collados, 2018](https://arxiv.org/abs/1808.09121)
  - Question an