I found the problem. When you create a new subprocess, pytorch sets the maximum number of potential useable threads by default (depends on how many cores in your current node). After I set_num_threads to 1 or 2, the speed behaves like what I expected.