Yeah, using the desktop with the 1080 was slower than the laptop with the 1080. I’m also using an SSD for both the desktop and laptop.
In any case, I examined the CPU usage of both the laptop and the desktop and saw the laptop under load was working at about 4.4 GHz for all CPUs while the desktop under load was staying at the minimum clock, 1.2 GHz for all CPUs (all this is training the with GPUs). This prompted me to check the CPU frequency ratios in the bios.
So I went into the bios and instead of using the default CPU frequency ratios of Auto, I manually set them to the advertised setting of (4.8 GHz) which seemed to fix things in PyTorch. And actually, when I was stress testing the CPUs under the Auto setting, I saw that the maximum it would achieve for an individual CPU is about 4.2GHz. So I went back and manually set the CPU frequency ratios to 4.1 GHz just as a test, and even here I saw a x2 speedup over the Auto setting.
So definitely something fishy is going on with the motherboard.
Also, what’s even weirder is the PyTorch code did get sped up, but the Tensorflow code did not. Pretty weird!
Thanks for your the suggestions though!