I would try to lower the number of variables a bit to further isolate the issue.
If I understand the use case correctly, @arrowtea has already used the same PyTorch, CUDA, cudnn versions on both machines. The only difference would be the hardware.
Could you run the test on the 1080 in your desktop and compare it to the 1080 in the laptop?