Bert Mesh Tensorflow
# 🌟 New model addition

## Model description

Bert Mesh Tensorflow is a modification of the original Bert that allows two important features:
1. Model parallelism.
2. Relative attention encoding.

## Open source status

* [X] the model implementation is available: (give details)
https://github.com/tensorflow/mesh/blob/master/mesh_tensorflow/bert/bert.py

* [x] the model weights are available: (give details)
A 1.8 billion pre-trained model for the language of life "Protein Sequences".

* [X] who are the authors: (mention them, if possible by @gh-username)
@adarob @sharannarang @nshazeer @toponado 

(Also tagging @LysandreJik  @patrickvonplaten @sgugger @julien-c @thomwolf  as they are the main contributors for Bert Model here in this repo)
Thanks a lot for open this @agemagician ! I think we should be able to adapt `modeling_bert.py` to be compatible with this model.
For model parallelism one could take a look at this PR: https://github.com/huggingface/transformers/pull/3578.
Regarding "relative attention encoding" does this just correspond to these lines: https://github.com/tensorflow/mesh/blob/d46ff8751f387cf37d732fa0fb968cc0d1de7cc2/mesh_tensorflow/bert/bert.py#L252 ? 

And could you also add a link to the weights here? 

I think it should be relatively straight-forward to implement this model. In a first step probably without model parallelism.
@agemagician also don't hesitate to give implementing it a try if you feel like it :-) <|||||>Thanks @patrickvonplaten for considering this modified Bert model.
I believe it is very important model as it allows new Bert SOT results with larger Bert models.
Furthermore, as your team familiar with T5 model it should be easy to integrate it, because most of this Bert model code overlap with T5 as both of them use mesh Tensorflow as a backend during training.

Regarding your questions/feedback:

1) For model parallelism one could take a look at this PR: #3578.
Yes, this is a good way to support model parallelism feature for all transformers models .
However, you could replicate the same T5 block code for Bert, if you think it will be faster:
https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_tf_t5.py#L541
later, you could implement a general modelling for model parallelism, but it is your call.

2) Regarding "relative attention encoding" does this just correspond to these lines: https://github.com/tensorflow/mesh/blob/d46ff8751f387cf37d732fa0fb968cc0d1de7cc2/mesh_tensorflow/bert/bert.py#L252 ?
Yes, this is correct, if I am not mistaken, it should be a replica of T5 code:
https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_tf_t5.py#L141

3) And could you also add a link to the weights here?
We didn't publish Bert-XL yet, as we still testing them.
I have sent you an email with private access to the model.
Later, we will publish them on our official repo:
https://github.com/agemagician/ProtTrans

4) One more difference that I have noticed is the "residual_structure" flag.
https://github.com/tensorflow/mesh/blob/ff0ef65f0ffb9c9c1d77564e63dd3ec2b9011436/mesh_tensorflow/bert/bert.py#L275
They either normalize the input before adding it to the previous layer output or after adding it. 

-----

PS: We want to thank your team in the acknowledgement section of our paper, for making our models more accessible to researchers: 
https://www.biorxiv.org/content/10.1101/2020.07.12.199554v2.full.pdf

Could you please let me know if the following draft format fits:
From Huggingface, we would like to deeply thank both Patrick and Julien for making the new T5 and Bert models easily accessible to researcher through transformers project.<|||||>Super will take a look soon :-) Thanks for sending me the weights! Regarding the acknowledgment, it would be very nice if you can include our transformers paper: https://arxiv.org/abs/1910.03771 and I guess a sentence like "We would like to deeply thank the HuggingFace team for making T5 and Bert accessible to researcher by means of transformers [Citation]" would be awesome :-)<|||||>Thanks a lot @patrickvonplaten for looking into this model.

Sure, it will be our great pleasure to include this acknowledgement into our next paper version.

Looking forward to hear good news from you soon.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>@patrickvonplaten any updates on implementing mesh Bert ?<|||||>@agemagician - didn't find time yet sadly.... :-/ Hope to have a bit more time in ~2 weeks. Don't hesitate to ping me on it though ;-) 
<|||||>Thanks, I will ping you after 2 weeks :)<|||||>@patrickvonplaten  Any update on this issue ?
We are over half way of finishing the training of a 2 Billion Bert model.
We are planing afterwards to train a 8 Billion Bert model, but we have to make sure that everything runs perfectly for the 2 Billion model using mesh tensorflow + huggingface.
<|||||>Having thought about this a bit more, we would have to add a new BertModel class for this. Given that this will be quite time-consuming (mesh tensorflow debugging) and that this is not an official and much-requested model, I don't think I'll be able to take time to add it. Sorry @agemagician! If it's very important for you, I think you will have to take a stab at it yourself (shouldn't be too difficult though)<|||||>Thanks @patrickvonplaten for your consideration.
In this case, I will start debugging and I will make a pull request if I made it work.

<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
