The logic is correct, what I am trying to do is to do self attention between 16 parts of an image. So I transpose the batch_size and 16. The multi head attention here implemented takes the (batch_size, seq_len, emb) shape. So I transpose to get (25,16,2048) as input shape. But Transposing doesn’t work, but without transposing works. Is there any method to check the gradient flow?