Solved by DeuS_CaNoN in post #6 
              
              
                I finally found out why I wasn’t getting a lot of utilization from my GPU. It looks like the Nvidia drivers has their own scheduler which automatically makes the parallel calls to their API single threaded. I did the below experiment 
void Test()
{
   int batchSize = 1;
   torch::Tensor randomTensor…