Not really; it slows down both the parallel and the nonparallel version of the code, but doesn’t result in the expected speedup of parallel vs nonparallel.
I have done some profiling using gprof. Program spends a lot of time in following three functions:
35% (of total):
std::_Sp_counted_base<(__gnu_cxx::Lock_policy)2>::M_release()
29%:
c10::intrusive_ptr<c10::TensorImpl, c10::UndefinedTensorImpl>::reset()
11%:
c10::intrusive_ptr<c10::TensorImpl, c10::detail::intrusive_target_default_null_type<<c10::TensorImpl >::reset()
Following code uses the torch model:
auto features = torch::empty({ 1,(long)(numFeats) });
		for (size_t i = 0; i < numFeats; i++)
		{
			features[0][i] = mdp.GetFeature(state, i);
		}
		//Note that NoGradGuard was called elsewhere
		torch::Tensor prediction = model->forward(features);			
		std::vector<float> logprobs(prediction.data<float>(), prediction.data<float>() + prediction.numel());

Any idea what could be causing this? Maybe the memory allocations/dealloacations? Is it possible to reuse the same memory for this?
Definition of model:
    torch::Tensor forward(torch::Tensor input) {
    {
	return torch::relu(linear3(torch::relu(linear2(torch::relu(linear1(input))))));
}
torch::nn::Linear linear1, linear2, linear3;