Return cross-attention from T5 models
Small change to the PyTorch and TF models to support returning cross-attention alignment scores. Previously it would only return self-attention alignment scores. This should solve the problem mentioned here: https://discuss.huggingface.co/t/how-to-get-cross-attention-values-of-t5/970

The code in this PR checks to see if the outputs of T5Block contains cross-attention, and if it does, it is returned from the T5Stack.
Hello,

Thank you for solution.

I just think the change will only return the cross-attention scores for the first layer of the decoder, since len(layer_outputs) will be bigger than 5 only in that case, (because has_relative_attention_bias will be true for i=0 and length of layer_outputs will be 6.)

        if output_attentions:
                all_attentions = all_attentions + (layer_outputs[2],)  # We keep only self-attention weights for now
                if len(layer_outputs) >= 5:
                    all_attentions = all_attentions + (layer_outputs[2],) + (layer_outputs[4],)
                else:
                    all_attentions = all_attentions + (layer_outputs[2],)

For the other layers of the decoder the len(layer_outputs) will be 4, so the cross-attention scores will be in 3rd index in those cases.

So I used the code like this, even I am not sure it's the nicest way to do this:

      if output_attentions:
                all_attentions = all_attentions + (layer_outputs[2],)  # We keep only self-attention weights for now
                if self.is_decoder:
                    if i==0:
                        all_attentions = all_attentions + (layer_outputs[4],)  # add cross-attention weights
                    else:
                        all_attentions = all_attentions + (layer_outputs[3],) # add cross-attention weights

<|||||>Thanks for catching that! With that fix we are able to get the cross-attention weights for each decoder layer.

It appears some tests need to be updated as well, but I'm not sure how best to do that.<|||||>Hi, any updates on this issue? Or at least some recommended quick fixes to the code that I can make directly to the code to get the cross-attention? For the fix proposed by @mg9, what exactly did you add to modeling_tf.py? (Do you mind sharing your updated modeling_tf.py?) With @mg9's fix, will the following code return the cross-attention? Thanks for your help in advance.

```
from transformers import T5Tokenizer, T5ForConditionalGeneration
tokenizer = T5Tokenizer.from_pretrained('t5-small')
input_ids = tokenizer("summarize: studies have shown that owning a dog is good for you ", return_tensors="pt").input_ids
decoder_input_ids = tokenizer("<pad>", add_special_tokens=False, return_tensors="pt").input_ids
model = T5ForConditionalGeneration.from_pretrained('t5-small', return_dict=True)
outputs = model(input_ids=input_ids, decoder_input_ids=decoder_input_ids, output_attentions=True)
outputs.encoder_attentions
```<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.