Sorry for not being clear enough. If you just want to time the CPU transfer time, call torch.cuda.synchronize() before starting and stopping the timer.
In your current code snippet you are starting the timer (, while potentially the some asynchronous CUDA calls are processed) then synchronizing, which will add the time of potential CUDA operations to the cpu() call.
To time the transfer time alone, use this:
# some operations
torch.cuda.synchronize()
start = time.time()
audio = audio.cpu()
torch.cuda.synchronize()
end = time.time()