Thanks for the response!
Just to clarify your comment about “occupying the GPU pretty well” – Basically, I found that when the batch size is small (e.g. 4), the GPU RAM is not fully utilized (<50% used at steady state). When I double my batch size to 8, the GPU RAM becomes almost 100% used, which makes sense since I’m doubling the batch size. However, I would have thought that this should indicate that I would get a 2x speedup, but it turns out that I don’t get that, more like 1.1x at best.
My guess is that perhaps there are other bottlenecks or the critical path in my model doesn’t actually benefit from an increased batch size, and so while peak memory usage does double, the bottlenecked portion isn’t able to use more memory to process things more quickly.