How do you time your code?
It contains a cudaMalloc which are known to be slow. This is why we actually have a custom cuda allocator in pytorch.
Also, I am not sure your launch parameters are optimal. You might want to do a bit more in each thread and not launch a million of them.