I finally found out why I wasn’t getting a lot of utilization from my GPU. It looks like the Nvidia drivers has their own scheduler which automatically makes the parallel calls to their API single threaded.

I would love to know where this is documented and whether it is possible to change this behavior. I’m using pytorch (Python) and I’ve never been able to get GPU utilization over 30%. I’ve played around with the number of worker threads, batch size, and so on. Nothing helps.
Is there example pytorch code I could look at that results in ~100% GPU usage to prove that this is even possible?