Pretraining T5-v1_1 on Flax
@patrickvonplaten 

In the[Flax tutorial](https://github.com/huggingface/transformers/tree/master/examples/flax/language-modeling) it is recommended loading the config from t5-v1_1-base when pretraining, using:

`config = T5Config.from_pretrained("google/t5-v1_1-base", vocab_size=tokenizer.get_vocab_size())`

This basically copies this [config](https://huggingface.co/google/t5-v1_1-base/blob/main/config.json)

It seems like this is tuned for finetuning, since it has the line:
'"dropout_rate": 0.1'

Google [states](https://huggingface.co/google/t5-v1_1-base) that _"Dropout was turned off in pre-training (quality win). Dropout should be re-enabled during fine-tuning."._ Should this be modified for pretraining?

Google also state that "no parameter sharing between embedding and classifier layer". How is this achieved?

Think both using `dropout_rate: 0.1` and not using it is fine! It also depends on the dataset you are using. The Flax T5 demo trains on Norwegian which is much smaller than English so it makes more sense here to use dropout for regularization<|||||>OK. Thanks. Im training a much larger model here, just using the Flax T5 Demo as a starting point. But if I understand you correctly, just simply changing this manually to  'dropout_rate: 0', would then be more in line with what Google describes in v1.1 - and then changing it back before finetuning. 

What about the change that is called _"no parameter sharing between embedding and classifier layer"_ that T5 v1.1 is using? I was unable to see how this is implemented in the example code. Is this a setting in config.json, or does it require changing the architecture.
<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.