T5: relative position embeddings
Hello, 
In the paper [Exploring the limits of Transformer Learning with a Unified Text-to-Text Transformer](https://arxiv.org/pdf/1910.10683.pdf) it says that they share the position embedding parameters across all layers.
<img width="967" alt="Screen Shot 2021-09-03 at 12 35 38 PM" src="https://user-images.githubusercontent.com/19749157/131946859-b23e5ff6-a1ae-4295-8511-4dc8fe273f7f.png">


But as shown below, the current implementation seems to use relative position embedding in the first layer only.
https://github.com/huggingface/transformers/blob/c1c2d68d37d4d8372729038ca06246e31859beaa/src/transformers/models/t5/modeling_t5.py#L807

I'm not sure how it changes the performance, but are there any particular reason for such implementation?
Hi @hyukyu ,

T5 implementation does share the relative embedddings. The `has_relative_attention_bias` specifies which layer should store the embedding, which is the first layer in T5. Then the `position_bias` is computed in the first layer and shared with all other layers. 

See
https://github.com/huggingface/transformers/blob/76c4d8bf26de3e4ab23b8afeed68479c2bbd9cbd/src/transformers/models/t5/modeling_t5.py#L487-L496

https://github.com/huggingface/transformers/blob/76c4d8bf26de3e4ab23b8afeed68479c2bbd9cbd/src/transformers/models/t5/modeling_t5.py#L1019-L1022<|||||>Hello @patil-suraj,

I misunderstood the code, but now it is very clear that they do use relative embeddings in the other layers too.

I really appreciate your help. Thank you!