KFrank:

permute(0,2,1),dim=2)


Thanks Frank for the reply.
Sorry that was my bad, in wpe[w][:] = torch.sum(atten_matrix[B][:][i] * ee.permute(0,2,1),dim=2) w should be i. About wpe, atten_matrix[B][:][i] * ee.permute(0,2,1) gives me L vectors (L x dim) and I sum them up along the second (actually third) dimension (dim=2), which gives me one vector (1 x dim).