Maybe it has something to do with CUDA version?
My environment is CUDA 10.1 on both the Linux and Windows machines mentioned.
I just ran a quick test on CUDA 11.2 and it seems to perform okay without torch.cuda.synchronize().
Could you confirm you are not running 10.1?
Thanks.
If this is indeed the issue, then I think I could close the thread.