Did you make sure to synchronize the code properly before starting and stopping the timer?
Since CUDA operations are asynchronous, you’ll get wrong profiling results without synchronizations.
Also, are you using the same PyTorch, CUDA and cudnn versions?