Briefly, you need either multiple torch.cuda.stream contexts, or jit.fork (in jit compiled code) to also separate cpu threads (that enqueue cuda operations). Unfortunately, speedup on one gpu may be limited, if gpu utilization in affected code is already high, or code fragments are small.