You mean that our users do not need do anything for the overlap?

No! The CPU will only wait for the GPU to finish computing things when the values are actually needed on the CPU side (when you try to print a Tensor, send it to cpu wit .cpu() or get the content as a python number with .item(), etc).
You can play with this by creating a cuda Tensor and apply simple operations on it and use time.time() to see what takes time (I did not run the code as I don’t have a GPU on this machine, so there might be typo):
import torch
from torch.cuda import synchronize
from time import time

small_tensor = torch.rand(10, device="cuda")
medium_tensor = torch.rand(1000, device="cuda")
# I used random values for the size here, there might need to be bigger for you
# to see a significant difference depending on your GPU
large_tensor = torch.rand(100000, device="cuda")
# Make sure all is properly finished on the GPU side
synchronize()

start = time()
# Do an inplace op to avoid any output allocation overhead
small_tensor.add_(1)
print("Small no sync ", time() - start)

start = time()
# Do an inplace op to avoid any output allocation overhead
medium_tensor.add_(1)
print("Medium no sync ", time() - start)

start = time()
# Do an inplace op to avoid any output allocation overhead
large_tensor.add_(1)
print("Large no sync ", time() - start)
print("All of these should take more or less the same time to run")
print("even if you increase the size a LOT")
print("This basically just measure how long it takes to queue the task on the GPU")


synchronize()
start = time()
# Do an inplace op to avoid any output allocation overhead
small_tensor.add_(1)
synchronize()
print("Small no sync ", time() - start)

synchronize()
start = time()
# Do an inplace op to avoid any output allocation overhead
medium_tensor.add_(1)
synchronize()
print("Medium no sync ", time() - start)

synchronize()
start = time()
# Do an inplace op to avoid any output allocation overhead
large_tensor.add_(1)
synchronize()
print("Large no sync ", time() - start)
print("In this one, you should see the runtime actually going up with the size!")

As a bonus, you can run that with the env variable “CUDA_LAUNCH_BLOCKING=1”. It will force the whole cuda api to be synchronous. And so if you set this, you should see similar results both with and without synchronization.