The cuda api is asynchronous. so yes it queues tasks for the gpu to execute. torch.cuda.synchronize() simply waits for all outstanding tasks to be finished before returning.
Async copy will be async. It depends on your workload/code whether or not it’s going to be faster. But it won’t make a huge difference for sure.