Sine:

[[[1,2,3,4,0,0,0],[2,3,5,0,0,0,0],[0,0,0,0,0,0,0]],[[3,2,3,4,0,0,0],[0,0,0,0,0,0,0],[0,0,0,0,0,0,0]]]


It would increase the computation. Apart from that, you may have to assign the logits of the padded entries to a higher negative value, so that they don’t contribute to attention probability.