I’m not sure how you are profiling your code, but note that CUDA operations are executed asynchronously, so you would need to synchronize the code via torch.cuda.synchronize() before starting and stopping the timer.
If your device is busy with other workload (e.g. the model training), the tensor copy operation might accumulate the previous timings, and your profiling would be invalid.