Yeah, here are some more details. My PyTorch version is torch==1.3.0+cu100. Here’s the script I’m running for inference:

  
      gist.github.com
  
  
    https://gist.github.com/bfortuner/c99a32dbdeb0f4f560d0080118f15af8 16
cpp-inference.cpp
#include <torch/script.h> // One-stop header.
#include <torch/torch.h>
#include <iostream>
#include <memory>

int main(int argc, const char* argv[]) {
  if (argc != 2) {
    std::cerr << "usage: example-app <path-to-exported-script-module>\n";
    return -1;
  }
This file has been truncated. show original




  
  
    
    
  
  


Then I run this
mkdir build && cd build
cp ../traced_resnet_model.pt .
cmake -DCMAKE_PREFIX_PATH=/home/bfortuner/pytorch_libs/libtorch ..
make
./cpp-inference traced_resnet_model.pt

If I download this version of libtorch: https://download.pytorch.org/libtorch/cu101/libtorch-shared-with-deps-1.3.0.zip 13, I’m able to run inference on CPU, but it doesn’t recognize my GPU (this makes sense bc the cuda versions aren’t matching).
If I download this version of libtorch: https://download.pytorch.org/libtorch/cu100/libtorch-shared-with-deps-1.3.0.zip 30, I’m able to do inference on CPU, but it still doesn’t recognize my GPU:
(.venvpy3) bfortuner@bfortuner-desktop:~/workplace/pytorch_spike/build$ ./cpp-inference ../traced_resnet_model.pt 
CUDA not available! Training on CPU.
ok
-0.0172 -0.5685  0.2170 -0.8681  0.3364
[ Variable[CPUFloatType]{1,5} ]

If I try to load the model onto GPU anyway, I get this:
CUDA not available! Training on CPU.
ok
terminate called after throwing an instance of 'c10::Error'
  what():  CUDA error: CUDA driver version is insufficient for CUDA runtime version (getDevice at /pytorch/c10/cuda/impl/CUDAGuardImpl.h:37)
frame #0: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x33 (0x7f99fa1fd813 in /home/bfortuner/pytorch_libs/libtorch/lib/libc10.so)
frame #1: <unknown function> + 0x1234b (0x7f99f846334b in /home/bfortuner/pytorch_libs/libtorch/lib/libc10_cuda.so)
frame #2: at::native::to(at::Tensor const&, c10::Device, c10::ScalarType, bool, bool) + 0x686 (0x7f99fbff5c96 in /home/bfortuner/pytorch_libs/libtorch/lib/libtorch.so)
frame #3: <unknown function> + 0x1efb7f1 (0x7f99fc31d7f1 in /home/bfortuner/pytorch_libs/libtorch/lib/libtorch.so)
frame #4: <unknown function> + 0x3a8c14b (0x7f99fdeae14b in /home/bfortuner/pytorch_libs/libtorch/lib/libtorch.so)
frame #5: <unknown function> + 0x40411d2 (0x7f99fe4631d2 in /home/bfortuner/pytorch_libs/libtorch/lib/libtorch.so)
frame #6: torch::jit::script::Module::to_impl(c10::optional<c10::Device> const&, c10::optional<c10::ScalarType> const&, bool) + 0x13e (0x7f99fe46730e in /home/bfortuner/pytorch_libs/libtorch/lib/libtorch.so)
frame #7: torch::jit::script::Module::to_impl(c10::optional<c10::Device> const&, c10::optional<c10::ScalarType> const&, bool) + 0xb0 (0x7f99fe467280 in /home/bfortuner/pytorch_libs/libtorch/lib/libtorch.so)
frame #8: torch::jit::script::Module::to(c10::Device, bool) + 0x29 (0x7f99fe4676a9 in /home/bfortuner/pytorch_libs/libtorch/lib/libtorch.so)
frame #9: main + 0x176 (0x407e7e in ./cpp-inference)
frame #10: __libc_start_main + 0xf5 (0x7f99f9477f45 in /lib/x86_64-linux-gnu/libc.so.6)
frame #11: ./cpp-inference() [0x407ba9]

If I download this version of libtorch: https://download.pytorch.org/libtorch/cu100/libtorch-shared-with-deps-latest.zip 16, it recognizes the GPU, but fails loading the model on CPU and GPU.
CUDA is available! Training on GPU.
terminate called after throwing an instance of 'c10::Error'
  what():  [enforce fail at inline_container.cc:137] . PytorchStreamReader failed closing reader: file not found
frame #0: c10::ThrowEnforceNotMet(char const*, int, char const*, std::string const&, void const*) + 0x47 (0x7fc2b7236e17 in /home/bfortuner/pytorch_libs/libtorch/lib/libc10.so)
frame #1: caffe2::serialize::PyTorchStreamReader::valid(char const*) + 0x6b (0x7fc2b9f944cb in /home/bfortuner/pytorch_libs/libtorch/lib/libtorch.so)
frame #2: caffe2::serialize::PyTorchStreamReader::~PyTorchStreamReader() + 0x1f (0x7fc2b9f9451f in /home/bfortuner/pytorch_libs/libtorch/lib/libtorch.so)
frame #3: <unknown function> + 0x3c13b97 (0x7fc2bb070b97 in /home/bfortuner/pytorch_libs/libtorch/lib/libtorch.so)
frame #4: torch::jit::load(std::unique_ptr<caffe2::serialize::ReadAdapterInterface, std::default_delete<caffe2::serialize::ReadAdapterInterface> >, c10::optional<c10::Device>, std::unordered_map<std::string, std::string, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, std::string> > >&) + 0x238 (0x7fc2bb077ba8 in /home/bfortuner/pytorch_libs/libtorch/lib/libtorch.so)
frame #5: torch::jit::load(std::string const&, c10::optional<c10::Device>, std::unordered_map<std::string, std::string, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, std::string> > >&) + 0x69 (0x7fc2bb077cd9 in /home/bfortuner/pytorch_libs/libtorch/lib/libtorch.so)
frame #6: main + 0x125 (0x4220ab in ./cpp-inference)
frame #7: __libc_start_main + 0xf5 (0x7fc2b64adf45 in /lib/x86_64-linux-gnu/libc.so.6)
frame #8: ./cpp-inference() [0x420e89]