lkp411:

I remember reading somewhere here that copying data from the gpu to the cpu forces a sync. Is this true? And does it still happen in a no_sync context?


Yes, this is true, since you would need the calculated tensor in order to push it to the CPU (thus the CPU has to synchronize on the GPU workload).
This should also be the case in a DDP setup with no_sync().