I don’t think so since this is a ‘‘partial’’ transpose on the 4th and 3rd dimension of the tensor
check
torch.matmul(Wqs, x.transpose(1,2)) -  torch.matmul(x, Wqs.transpose(3, 2)).transpose(3, 2)
is not a zero matrix