Indeed, the slower machine is using a hard drive, while the faster one uses an M.2 drive. However, my data comes from a simulator and is not loaded from any disks, so I don’t believe that is where the slowdown comes from.
Also, while there is indeed a difference in the CPUs, their usage is never maxed out, so I don’t really know if that’s the bottleneck. In any case, I run the same experiments in a server with an Intel Xeon Gold 5118 and a Tesla V100 and the same experiment runs in 28 hours.
I will try and profile my code in some of these machines to see if I find out what’s going on.