yuri:

ame oper


hi @yuri using _stacked = torch.matmul(x, Wqs.transpose(3, 2)).transpose(3, 2) also throws an assertion error. I am beginning to wonder if it might be an issue with numerical precision as the absolute value of the errors are in the range of 10^-7, some are even an order of magnitude lower. Either that or matmul might be doing something undocumented.