The big thing is that autograd will not split the graph to sample by sample basis when you use batch operations. That alone helps a ton, but sometimes cannot be done, I don’t know what your application is. I would start there and see if you still have the GPU throughput issue.