leimao:

I think only synchronize after the code block is fine.


If you are throwing out the first iteration, then synchronizing before stopping the timer might be working.
In any case, torch.utils.benchmark provides a safe approach to profile workloads, which I would generally recommend to void these issues coming from manual profiling.



 leimao:

One thing I found is that for large torch models, such as ResNet50, the latency measured from adding synchronize does not make too much difference to the one from without synchronize.


I don’t think the size of the model matters, but the general ability of the CPU to run ahead.
I.e. if the model contains a lot of (small) layers, data-dependent control flow etc. your profiling might not see a large difference between a synchronized and an async run. However, I usually don’t dig into profiles, which were created without syncs, as the majority is wrong.