yuri:

please be aware that the two operations are not the same. the second one is equivalent to passing through a linear layer. the one you posted is not


In my understanding, both operations achieve the same result. No?
Assuming A = Wqs, B = x, C = _stacked



 schow:

_stacked = torch.matmul(Wqs, x.transpose(1,2))


This says C = A.(B^T).



 schow:

_stacked = torch.matmul(x, Wqs.transpose(3, 2)).transpose(3, 2)


The change that @yuri suggested does C= (B.(A^T))^T = A.(B^T).