Details on T5's current integration status 
Hi all,

Regarding Google's T5 model, here is a quick summary of the status:
* the core model is in the library and some people have started to use it, but:
  - while the operations are identical or very similar (einsum vs. matmul), there is quite a significantly higher relative error between this model's PT hidden-state and the mesh-tensorflow hidden-state (in particular compared to our previous TF => PT model conversions).
  - our guess is that this comes for a combination of bfloat16 vs. fp32, einsum+model parallelism vs. matmul, plus the fact that we are not masking the hidden-states at each layer as the original implementation do (this should not matter much though).
  - as a consequence, we are waiting to be able to confirm it's performances on a GLUE fine-tuning before having a wider communication on its addition to the library.
* the full integration with GLUE tests requires a few features that we still need to add:
  - a decoding mechanism,
  - a pre/post-processing for GLUE to use it in text-to-text setting, and
  - a model parallelism feature
^^ we plan to work on these in February (from the more general view of having better encoder-decoder support in the library).

cc @julien-c @LysandreJik @sshleifer @patrickvonplaten
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
