I think this just returning the attentions from the deocder since the encoder hidden states are only computed once in the generate function. So the returned attentions at every iteration is attention over the tokens generated so far.
pinging @sshleifer