ptrblck:

If you are throwing out the first iteration, then synchronizing before stopping the timer might be working.
In any case, torch.utils.benchmark provides a safe approach to profile workloads, which I would generally recommend to void these issues coming from manual profiling.


Thanks. I was actually assuming there was no GPU activities or it has already been synchronized before the first iteration. Having the synchronization before the first iteration makes sense based on your description.
I will also check the benchmark tool.