Hello,
I am trying to finetune the T5 model and need to get cross attention scores as well as self-attention scores. However when I set output_attentions=True, the model only returns self-attention values.
Any idea how to get cross-attention values such as 6 elements with B,8,Tx,Ty ? (num_heads=8, num_layers=6)
I am doing forward call on t5 model
t5 = T5ForConditionalGeneration.from_pretrained(“t5-small”)
outputs = t5(input_ids=input_ids, labels=output_ids, use_cache=False, output_attentions=True, output_hidden_states=True)
output returns 7 elements
outputs[0]: loss
outputs[1]: input_embeddings
outputs[2]: decoder_hiddens :  B,Ty,H
outputs[3]: decoder_self_attentions     # B,8,Ty,Ty
outputs[4]: encoder_last_hidden states ?
outputs[5]: encoder_hiddens?
outputs[6]: encoder_self_attentions # B,Tx, Tx
Thanks a lot!