I think only synchronize after the code block is fine.
One thing I found is that for large torch models, such as ResNet50, the latency measured from adding synchronize does not make too much difference to the one from without synchronize. According to PyTorch semantics, PyTorch does implicit synchronization when there is memory copy. However, I actually don’t see there are memory copy in those models. So I suspect that there are some PyTorch ops used in the model that do synchronizations. Can you please provide some explanations or details? Thank you.