@azhx what shape are the attentions?
What’s going on is called “incremental decoding/caching”. Bart is reusing all the information it can from the previous decoder step and trying to only do “math” on the latest decoder token.
This all happens before attn_weights are computed (all done by https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_bart.py#L698 2), so I would expect your decoder self attention weights to be of shape ` (batch_size, num_heads, cur_len, cur_len) at each step.
Note that cross attention is never returned. I have a very experimental working branch that concatenates the cross attn outputs with the self attention outputs and returns them all as one big tensor: https://github.com/huggingface/transformers/pull/6967 2