Thanks all the same! I used torch.get_num_threads() to see # threads being used, and I found that was the cause of abnormal slow-down. When I set it back to OMP_NUM_THREADS=# cores, it came back to normal.
However, I was wondering:

What is the best practice for setting OMP_NUM_THREADS? # physical cores or logical cores? or neither?
Where is the best place to set it? I have two options:
2.1) set env vars OMP_NUM_THREADS when submitting jobs to cluster
2.2) set via torch.set_num_threads()
Not sure which way is better and why. Can you help?