I have a model of convolutional filters that I’m applying to images and 11GB of RAM on the GPU. Somehow, increasing batch size while still having things fit in memory doesn’t seem to improve the speed that much.
When I do training with batch size 2, it takes something like 1.5s per batch. If I increase it to batch size 8, the training loop now takes 4.7s per batch, so only a 1.3x speedup instead of 4x speedup.
This is also true for evaluation. Evaluating batch size 1 takes 0.04s, but batch size 4 takes 0.12s, batch size 8 takes 0.24s.
Is it reasonable to expect that increasing batch size should allow me to utilize more of the GPU RAM and hence get a speedup almost for free?
What could be things I should look at? For one, I’ve tried to make sure the tensors are contiguous in memory (so I avoid having to call .contiguous()).