The first CUDA operation creates the CUDA context on the device, which adds some overhead.
Also, since CUDA operations are asynchronously called, you would need to synchronize the code before starting and stopping a timer via torch.cuda.synchronize().
Additionally, if you are using torch.backends.cudnn.benchmark = True the first iteration using a new input shape will profile different cudnn kernels and select the fastest one, which also adds overhead.