t511) improvements compared to the original T5 model during the pretraining: - GEGLU activation in feed-forward hidden layer, rather than ReLU - see [here](https://arxiv.org/abs/2002.05202) - Dropout was turned off in pretraining (quality win). Dropout should be re-enabled during fine-tuning - Pretrained on span-based masked language modeling (MLM) objective only without mixing in the downstream tasks - No parameter sharing between embedding and classifier layer  This model also used the "efficient" T5 architecture findings presented in [this paper](https://arxiv.org/abs/2109.10686). In a nutshell, the paper indicates that a Deep-Narrow model architecture is favorable for downstream performance compared to other model architectures of similar parameter count. To be more precise, model depth is defined as the number of transformer blocks that are stacked sequentially.  This model uses the [t5-efficient-large-nl36](https://huggingface.co/google/t5-efficient-large-nl36) architecture's layer depth which means both the encoder and the decoder have 36 transformer layers compared to the original T5 "large" model's architecture of 24 transformer layers.  In total, this model has 1425 million parameters.  