Hi, if I have understood this correctly, the problem you are facing is due to your data i/o being slower than the gpu ops. This could be happening due to a variety of reasons, including expensive read operation due to on the fly decompression, slow storage device, other programs using up cpu resources. You could try to check the speed of your data i/o without passing any data to your model, to see if it stays constant across batch sizes.
Hope this gives you the right direction to solve this problem.