Does this 2x perf difference make sense?
In [4]: %timeit torch.mm(torch.Tensor(512, 20).cuda(), torch.Tensor(512, 20).cuda().t())
The slowest run took 31073.84 times longer than the fastest. This could mean that an intermediate result is being cached.
1 loop, best of 3: 78.9 µs per loop

In [5]: %timeit torch.bmm(torch.Tensor(20, 512, 1).cuda(), torch.Tensor(20, 512, 1).cuda().transpose(-1, -2))
The slowest run took 64.09 times longer than the fastest. This could mean that an intermediate result is being cached.
10000 loops, best of 3: 128 µs per loop

Thanks!