Finally I find the main reason is that I use the libtorch built from source by myself, which is slower than official libtorch significantly.
Is there an official guide about how to build libtorch from source?