I think most of the time is spent sending data to the GPU, instead of calculating multiplication on GPU.
import torch

a = torch.randn(1000000, 3)
b = torch.randn(1000000, 3)

with torch.autograd.profiler.profile(use_cuda=True) as prof:
    torch.mm(a.T, b)
print(prof.key_averages().table(sort_by="self_cpu_time_total"))
# Self CPU time total: 12.777ms
# CUDA time total: 12.951ms

a, b = a.cuda(), b.cuda()
with torch.autograd.profiler.profile(use_cuda=True) as prof:
    torch.mm(a.T, b)
print(prof.key_averages().table(sort_by="self_cpu_time_total"))
# Self CPU time total: 1.389ms
# CUDA time total: 1.8451ms

I use RTX 2080ti and Xeon® E5-2678 v3