To add to Alban’s comments:
If you want just avoid the C++ API for NNs to be included, there is the NO_API cmake flag and there is (but now you’re in the danger zone) INTERN_DISABLE_AUTOGRAD, but I must admit I haven’t experimented with it.
If you want to remove more, you might want to dig into the built of PyTorch (personally, I find this more daunting than the C++ code, but this could be me). The little-known and perhaps not completely intuitive detail is that the libtorch build is currently (so if you read this in late 2021 or 2022 it might have changed) defined in the caffe2/CMakeList.txt. More specifically, this is the place where the Torch bits (C++ API, Autograd, JIT, …) get added to libtorch.so:

  
      github.com
  
  
    pytorch/pytorch/blob/e606e603312cf874127b560bcbb8b78b9574ac84/caffe2/CMakeLists.txt#L614 1

      ${TORCH_SRC_DIR}/csrc/api/src/optim/lbfgs.cpp      ${TORCH_SRC_DIR}/csrc/api/src/optim/optimizer.cpp      ${TORCH_SRC_DIR}/csrc/api/src/optim/rmsprop.cpp      ${TORCH_SRC_DIR}/csrc/api/src/optim/serialize.cpp      ${TORCH_SRC_DIR}/csrc/api/src/optim/sgd.cpp      ${TORCH_SRC_DIR}/csrc/api/src/serialize/input-archive.cpp      ${TORCH_SRC_DIR}/csrc/api/src/serialize/output-archive.cpp    )  endif()  list(APPEND Caffe2_CPU_SRCS ${TORCH_SRCS})endif()# NOTE [ Linking AVX and non-AVX files ]## Regardless of the CPU capabilities, we build some files with AVX and AVX2# instruction set. If the host CPU doesn't support those, we simply ignore their# functions at runtime during dispatch.## We must make sure that those files are at the end of the input list when# linking the torch_cpu library. Otherwise, the following error scenario might



  
  
    
    
  
  


For a quick test, you could see to changing that, but it probably is a good idea to not call the result libtorch if you plan on doing things with it.
Whether or not I would actually recommend to do such a thing is on another page, you’re clearly in “use at your own peril”-land here.
Best regards
Thomas