Hi,
Thanks for the clear description.
The CUDA is asynchronous for execution by default. So it is already very likely that a lot of overlap is happening.
Note that you should use torch.cuda.synchronize() if you record times on the CPU to make sure you measure the proper time!