I finally found out why I wasn’t getting a lot of utilization from my GPU. It looks like the Nvidia drivers has their own scheduler which automatically makes the parallel calls to their API single threaded. I did the below experiment
void Test()
{
   int batchSize = 1;
   torch::Tensor randomTensor = torch::rand({batchSize, 4, 15, 15})
   while (true)
   {
        network->forward(randomTensor);
   }
}

int numThreads = 5;
std::vector<std::thread>> threads;
for (int i = 0; i < numThreads; i++)
{
    threads.push_back(std::thread(Test));
}
for (int i = 0; i < numThreads; i++)
{
    threads[i].join();
}

No matter how much I increased my numThread variable, it kept at 30% utilization, but the second I increased batchSize from 1 to 10, the utilization jumped from 30% to 90%. What’s even more interesting is that it will keep using 90% even if numThreads == 1.
This doesn’t help me because in my training, I’m way more often only curious in 1-4 different inputs, but hopefully this can shed some light to any future developers wondering why GPU utilization might be low.