Overview
T5v1.1 was released in the google-research/text-to-text-transfer-transformer
repository by Colin Raffel et al. It's an improved version of the original T5 model.
One can directly plug in the weights of T5v1.1 into a T5 model, like so:
```python
from transformers import T5ForConditionalGeneration
model = T5ForConditionalGeneration.from_pretrained("google/t5-v1_1-base")
```
T5 Version 1.1 includes the following improvements compared to the original T5 model:
GEGLU activation in the feed-forward hidden layer, rather than ReLU. See this paper.
Dropout was turned off in pre-training (quality win). Dropout should be re-enabled during fine-tuning.
Pre-trained on C4 only without mixing in the downstream tasks.
No parameter sharing between the embedding and classifier layer.
"xl" and "xxl" replace "3B" and "11B". The model shapes are a bit different - larger d_model and smaller
