The first CUDA call initializes the CUDA context, which is this slow.
You might want to add an init method, which calls into torch.cuda.init() or any random CUDA tensor creation.
Also note, that you are not timing the tensor creation in this current code snippet, since you are not synchronizing the code. The 1e-6 times might thus profile the Python overhead from the kernel launch.