After fighting few hours, I finally figure out the trouble with the static library of pytorch.
The main trouble was the --force-load or --whole-archive need to be apply on some specific library not only torch but not all of them.
Now I am able to run libtorch with and without Cuda over:
OSX Catalina (CPU)
OSX High Sierra (CPU/GPU)
Ubuntu 18.04 (CPU/GPU)
I am facing another issue now on Windows. The build failed because of this error :
error C2338: CopyToCPU requires fundamental types
If someone has an idea how to fix it, that will be very nice.
Thanks