The issue is because of -D_GLIBCXX_USE_CXX11_ABI=0 flag. Here is the related disscusion 4. Everything is fine when I download the [libtorch-cxx11-abi-shared-without-deps-latest.zip] version.
But the conflict between Arrow and libtorch is still there.