Are you providing any CUDA source files or use with_cuda=True?
I just tested a simple C++ extension on my non-GPU machine and it worked fine:
cpp_source = '''
torch::Tensor my_fun(torch::Tensor z) {
  auto s = torch::relu(z);
  return s;
}
'''

module = load_inline(name='inline_extension',
                     cpp_sources=[cpp_source],
                     functions=['my_fun'])

module.my_fun(torch.randn(10))