T5 relative attention bias: Discrepancy to original implementation
### Who can help
@patrickvonplaten 

## Information

Model I am using: T5

In the huggingface TF T5 implementation, the relative attention bias only seems to be applied to the first layer of the stack. If I understand the original implementation correctly, though, it is applied to all layers there.

HF: https://github.com/huggingface/transformers/blob/master/src/transformers/models/t5/modeling_tf_t5.py#L570
Mesh: https://github.com/tensorflow/mesh/blob/master/mesh_tensorflow/transformer/transformer_layers.py#L263
The implementation of T5 is based on the original implementation which can be found [here](https://github.com/google-research/text-to-text-transfer-transformer).

The implementation you are referring to is a general Transformer implementation from Tensorflow Mesh. It seems like this repo does not implement T5, but other models like the Funnel Transformer and the Evolved Transformer.

<|||||>But the T5 repo actually calls the Tensorflow Mesh implementation internally. AFAIK there is no standalone t5 implementation (apart from the HF one).<|||||>Ok I see. In that case I'll leave it on to Patrick to help you. <|||||>Hey @maurice-g, we compute it ones and then forward it to all the follow-up layers since the result will be the same for all layers. 2 month ago, I made sure that our implementation is exactly the same as the original implementation - see those tests: https://github.com/huggingface/transformers/blob/e94d63f6cbf5efe288e41d9840a96a5857090617/tests/test_modeling_t5.py#L734 and https://github.com/huggingface/transformers/blob/e94d63f6cbf5efe288e41d9840a96a5857090617/tests/test_modeling_tf_t5.py#L492 
So I'm confident that the model behaves as expected, but in case you have a reproducible code snippet showcasing differences between the 2 implementations, I'm more than happy to take a look :-)<|||||>You're right, missed that they were returned and forwarded to the other layers. Sorry for that.<|||||>No worries! Thanks for checking in-detail. I think it's always a very good practice to check things in-detail or more often than not you will find subtle bugs in Transformers that will help us improve the code :-)