This is expected, since the first iteration would initialize the CUDA context etc. and is thus slow.
Also, to properly profile CUDA operations, you would have to synchronize the code manually, since CUDA ops are executed asynchronously, which is why we recommend to use torch.utils.benchmark to profile workloads (which adds warmup iterations, synchronizes, and averages the runtimes).