If you have a Xeon server with 48 hyperthreads (2 sockets, 24 physical cores), you will almost surely have worse perf by default than i7 with 8 cores.
The reason here is that Intel MKL and Intel OpenMP which PyTorch uses to parallelize it’s CPU code, try to use all possible cores by default.
However, using all cores is not great in many cases. And its especially not great when you have 12 cores split across 2 sockets, and oversubscribed with hyperthreads.
Try to run your python program like this on your Linux machine:
OMP_NUM_THREADS=8 MKL_NUM_THREADS=8 python foo.py

Try different values than 8 from 1 to 12. Going more than 12 will surely be bad perf, unless you have very large workloads.