@albanD Thanks for response !  could you please elaborate more? I’ve seen elsewhere that pytorch just asynchronously (queues?) adds operations to gpu. If that’s the case what does torch.cuda.synchronize() do exactly (and how do I speed it up with async copy)? When I measure time with time.time() and with torch.cuda.synchronize() before t1 and t2, it says its very fast.
EDIT: Async copy should make it faster? I have tried pin_memory=True on the dataloader and .to(non_blocking=True), if async copy is the way, is that all ?