GabbyChan:

Is that just normally how Pytorch would work?


It depends on the operations you are using. If the current method synchronizes the code by e.g. pushing a value to the CPU, then you could see synchronizations in the timeline.
The majority of ops in PyTorch do not synchronize besides a few exceptions (e.g. some methods in the torch.linalg namespace check the output info to give you an error in case the linalg operation failed. This should be relaxed in upcoming PyTorch releases).



 GabbyChan:

Does the runtime from the profiler representing the GPU and CPU runtime …?


Yes, you would see the runtimes in the profile. Note that the profiling itself might add overhead.