Three quick comments:

You absolutely need to add torch.cuda.synchronize() to time cuda ops.
While NumPy does have an edge over PyTorch on the CPU for some operations, in this specific example I’d imagine that you have NumPy compiled with a better blas than PyTorch or something. For me, the timings are very similar, with PyTorch having an edge.
Gross slowness of PyTorch over NumPy is probably worth to file as a bug report. (My personal limit is 5-10 or so, but others are more ambitious.)

Best regards
Thomas