In case you are using the GPU, you would have to synchronize the code before starting and stopping the timers since CUDA operations are executed asynchronously. The better way to profile such workloads would be to use torch.utils.benchmark.