That code will likely be partly (perhaps mostly) measuring the time taken to do the actual processing in waveglow.infer which would of course be expected to be faster on the V100. To measure just the copy you should synchronize before  the copy, start timing, copy and them sync again before taking the end time. As in:
start = time.time()
torch.cuda.synchronize()
audio = audio.cpu()
torch.cuda.synchronize()
elapsed = time.time() - start

Or you can use CUDA events to do the timing. Here you’d do:
start = torch.cuda.Event(enable_timing=True)
end = torch.cuda.Event(enable_timing=True)
start.record()
audio = audio.cpu()
end.record()
torch.cuda.synchronize()
elapsed = start.elapsed_time(end)

The synchronize here is needed to retrieve the time, it is not actually related to timing and can be done at a later point. So you don’t have to slow down the processing with  synchronize by collecting timing.



 terryyizhong:

I tried use copy_() like:


Oh, I forgot to note that asynchronous copies can only happen on a non-default stream, they won’t work on the default stream 0. By default all work in PyTorch occurs on the stream 0 so asynchronous copies won’t happen. If you use a non-default stream you also need to synchronize it to the default stream yourself, by, for example recording an event on the default stream after issuing the work (i.e. calling the appropriate tensor methods, remembering they are asynchronous so work is not completed immediately), and then having the non-default stream wait on that event before copying. I won’t give an example as you need to understand the issues around synchronisation before applying this. There are examples of at least the PyTorch specific parts of this, though they generally assume some familiarity with CUDA programming.
I believe you also need to pin the CPU memory you are copying to, either by calling pin_memory on an existing tensor, or by passing pin_memory when creating the tensor (though I’m not sure all methods of creation support this parameter).
And, as noted asynchronous copies from GPU to CPU are only fixed in PyTorch 1.2 so unless you have upgraded they won’t work.
I can confirm that if properly implemented they can help, I reduced the impact of some work in a forward hook (so slowing down the training loop) by taking care of all the above issues (non-default stream/pin-memory/synchronise streams/PyTorch 1.2). But they won’t always help. If you go on to access items in the CPU tensor immediately after the copy then it will force synchronisation and performance won’t increase. But I think just issuing PyTorch operations on the CPU tensor is fine as they are asynchronous (again happy to be corrected by someone more knowledgeable here).