Hi, thanks a lot for your reply! But according to this post How to trace jit in evaluation mode using multi-gpu learned model? 5, if my model is generated in Python using torch.jit.trace().save(), will it still work to load the model and parallelize it in C++? (Maybe I mixed those concepts…)
Furthermore, I tried so but keep encountering data type errors, may I ask if the following is what you mean?
torch::jit::Module model = torch::jit::load(modelPath.c_str());
input_tensor = input_tensor.to(at::kCUDA).permute({0, 3, 1, 2}).contiguous();
torch::Tensor outputs = torch::nn::parallel::data_parallel(model, input_tensor);

But I encountered the error saying that
libtorch/include/torch/csrc/api/include/torch/nn/parallel/data_parallel.h:267:5: error: base operand of ‘->’ has non-pointer type ‘torch::jit::Module’
     module->to(devices->front());
     ^~~~~~
libtorch/include/torch/csrc/api/include/torch/nn/parallel/data_parallel.h:269:12: error: base operand of ‘->’ has non-pointer type ‘torch::jit::Module’
     return module->forward(std::move(input)).to(*output_device);
            ^~~~~~
libtorch/include/torch/csrc/api/include/torch/nn/parallel/data_parallel.h:279:28: error: no matching function for call to ‘replicate(torch::jit::Module&, std::vector<c10::Device>&)’
   auto replicas = replicate(module, *devices);
                   ~~~~~~~~~^~~~~~~~~~~~~~~~~~
libtorch/include/torch/csrc/api/include/torch/nn/parallel/data_parallel.h:142:42: note: candidate: template<class ModuleType> std::vector<std::shared_ptr<_Tp> > torch::nn::parallel::replicate(const std::shared_ptr<_Tp>&, const std::vector<c10::Device>&)
 std::vector<std::shared_ptr<ModuleType>> replicate(
                                          ^~~~~~~~~
libtorch/include/torch/csrc/api/include/torch/nn/parallel/data_parallel.h:142:42: note:   template argument deduction/substitution failed:
libtorch/include/torch/csrc/api/include/torch/nn/parallel/data_parallel.h:279:28: note:   ‘torch::jit::Module’ is not derived from ‘const std::shared_ptr<_Tp>’
   auto replicas = replicate(module, *devices);
                   ~~~~~~~~~^~~~~~~~~~~~~~~~~~
libtorch/include/torch/csrc/api/include/torch/nn/parallel/data_parallel.h:161:39: note: candidate: template<class ModuleType> std::vector<torch::nn::ModuleHolder<ModuleType> > torch::nn::parallel::replicate(const torch::nn::ModuleHolder<ModuleType>&, const std::vector<c10::Device>&)
 std::vector<ModuleHolder<ModuleType>> replicate(
                                       ^~~~~~~~~
libtorch/include/torch/csrc/api/include/torch/nn/parallel/data_parallel.h:161:39: note:   template argument deduction/substitution failed:
libtorch/include/torch/csrc/api/include/torch/nn/parallel/data_parallel.h:279:28: note:   ‘torch::jit::Module’ is not derived from ‘const torch::nn::ModuleHolder<ModuleType>’
   auto replicas = replicate(module, *devices);
                   ~~~~~~~~~^~~~~~~~~~~~~~~~~~

I tried to use _net = std::make_shared<torch::jit::Module>(model); or torch::Tensor outputs = torch::nn::parallel::data_parallel(&model, input_tensor); but none of them work.
Do I misunderstand what you said?