PyTorch uses MKL-DNN (https://github.com/intel/mkl-dnn 21) for CPU convolutions. It’s optimized 7 for Haswell and newer architectures (circa 2013+). I’ve never tried it on much older processors.
Two suggestions:

Run your program under perf top or perf record to see where the time is spent
Try adjusting OMP_NUM_THREADS. Try setting it to 1 or the number of unused cores (and values in between). Sometimes oversubscription (too many threads) can be a problem.