Thanks for helping!
So from what you’re saying, attn_weights are not necessary for this decoding step? I thought they were used compute the output of the attention layers in decoder blocks, which are part of the math required in the decoding step and so would need to be computed before the tokens are fed through?
I also thought the dimensions would be (batch_size, num_heads, cur_len, cur_len) but in one of the examples I tried, the attention weights shape seemed to be
(batch_size, num heads, 1 , cur_len)
That dimension is always one. For instance, the final index (index 37) of the decoder_attentions list in my example gave dimensions (8, 16, 1, 37). I suppose 8 (the batch_size) is due to the number of beams, 16 is clearly the num heads, and the final dimension corresponds with the index of the list (so cur_len) but I don’t understand the 1. The other trouble I mentioned was that 37 was not the length of the final best sequence prediction returned by generate() - that was 23.
@sshleifer