I forgot to leave a few notes here, just for the reference of other people (who might have the same doubts).
On macOS, here is the config of pytorch,
In [1]: print(th.__config__.parallel_info())
ATen/Parallel:
	at::get_num_threads() : 8
	at::get_num_interop_threads() : 8
OpenMP not found
Intel(R) Math Kernel Library Version 2019.0.4 Product Build 20190411 for Intel(R) 64 architecture applications
	mkl_get_max_threads() : 1
Intel(R) MKL-DNN v1.7.0 (Git Hash 7aed236906b1f7a05c0917e5257a1af05e9ff683)
std::thread::hardware_concurrency() : 16
Environment variables:
	OMP_NUM_THREADS : [not set]
	MKL_NUM_THREADS : [not set]
ATen parallel backend: native thread pool

OpenMP is not found for macOS pytorch, and by default get_num_threads=8.
By setting MKL_NUM_THREADS=16, the get_num_threads can be changed as well, since macOS pytorch is using MKL.
So, by setting MKL_NUM_THREADS, more CPU could be utilized.