There is some inherent windows problem with either L2 cache or memory page management, this difference is not specific to pytorch. Also, your builds may be not identical in regards to MKL, avx, openmp threading.