I just delete everything and rebuild from scratch, it seems I am able to build and install Libtorch now.
It may be a trouble when I was doing the submodule update --init --recursive.
Thanks for you help.