Although it is true that in single threaded mode, regular CPUs are often faster (higher clock speeds), the difference in your example is very large. I tested locally (i7-8700k @ 4.5GHz; 1.01s) and on our server (Xeon Gold 6142 CPU @ 2.60GHz; 2.16s) and the difference is not at all that noticeable. Some things you can check:

same version of PyTorch and Python (should not matter, but might indicate a version bug)
make sure the server is not busy (check htop for instance)
if you system is doing a lot IO (reading/writing) at the time that you run the benchmark, that can also negatively impact your CPU throughput