If you set pin_memory=True in your DataLoader the data in your Dataset will be loaded into pinned host memory, which makes the transfer to the device faster. Have a look at NVIDIA’s blog post 75 about this topic.
I’m not that sure that calc_loss really takes that much time on the CPU.
Since you are not synchronizing explicitly and call loss.data.cpu().numpy() this line of code has to wait for the GPU to finish all operations on loss, such that it can be transferred onto the CPU.
I guess the time just shows the waiting time for the GPU.
How did you time your script? Is the first profiling output using pin_memory=True while the second one doesn’t use it?