I got the same problem. I test same step in V100 and P100 card.
On V100 card machine, the .cpu() cost less than 0.01s.
But on P100 card machine, this step cost 5 second at most!
@ptrblck is that only about the GPU? the pytorch version and Cuda version are both same: CUDA 9 and pytorch 1.0.0