Replying to myself here - I did see a discussion on the older Torch issue trackers that their backend did exactly this (sync CUDA) at free, unless the caching memory allocator was enabled. I’m not sure how that translates to pyTorch, if it uses its own memory allocator by default instead of cudas and so get around this problem.