Problems with TFT5
As far as we are concerned, Tensorflow T5 has some problems:

- Shifting inputs right should add at the beginning of the sequences a *Start of Sequence token* (padding token in T5 case), but currently, it generates a useless sequence of zeros. This is due to this line of code:

```python
shifted_input_ids = tf.zeros_like(input_ids, dtype=tf.int32) # No matter what, this generates zeros
# ... carry this 0s all over the function
```

*Note*

We observe, that PyTorch's `T5ForConditionalGeneration` loss is a scalar. Contrary, with the `TFT5ForConditionalGeneration`, the loss is a tensor where each loss element corresponds to a *non-ignored* label. We think that the loss should be reduced. Is there any reason for keeping the loss *as is* instead of reducing it? 


Closing in favor of #6214 