problem with using past_key_values in T5ForConditionalGeneration
Hello! Here's my problem, I'm trying to add some modifications to past_key_values(like adding some tunable parameters) during training. To do this, I must pass both the _past_key_values_ and _labels_ parameters during training. I have implemented this in the BARTForConditionalGeneration model, but when I'm trying to do this in T5ForConditionalGeneration,  I find the following limitations.

In T5ForConditionalGeneration, when training, we're not able to use the parameter _past_key_values_ together with _labels_. 

And even if we use decoder_input_ids rather than labels, only the last target token is used, if the _past_key_values_ parameter is provided.

https://github.com/huggingface/transformers/blob/62832c962f85b5a554ebf8b930d13b76b9028a8d/src/transformers/models/t5/modeling_t5.py#L1596-L1603

However,  this limitation doesn't exist in other models like BART (which I think has a similar structure with T5).
https://github.com/huggingface/transformers/blob/62832c962f85b5a554ebf8b930d13b76b9028a8d/src/transformers/models/bart/modeling_bart.py#L1293-L1298

As far as I can see, the limitition in T5ForConditionalGeneration is not necessary.  Even if we pass both the _past_key_values_ and the _labels_, there'll be no errors since the inner modules like T5Block and T5Attention can handle the past_key_values properly(I'm trying to delete this limitation in T5 in my own fork, but I've not experimented yet, so I'm not 100% sure). Or are there some other purposes of this limitation in T5?
  

cc @patrickvonplaten <|||||>Hey @yssjtu, 

I agree with you! Would you like to open a PR to fix this? :-)