I was checking execution time of 5 epochs and it seemed weird that when I was trying to use all of my gpu memory, it was actually working twice slower than 1/4 max batch size. I do not mean it was learning slower, just the execution time of epoch. I dont know is that even an error, as far as I understand larger batches means squeezing more samples and averaging their losses to backtrack all of them at once, so theoretically larger batch should leads to shorten execution time. And batch_size=1 needs actually more time to do one epoch than batch_size=32, but although i have more memory in gpu the more I increase batch size from some point, the more its slowing down. I’m worried its because my hardware or some problem in code and Im not sure should it works like that.
Here are some of my results during ShuffleNet V2 x0.5 training:
Fine tuning / num_workers: 4 / Optimizer: SGD / Loss: Cross Entropy /  Data Augmentation: Rotate, Horizontal and Vertical Flip / Outputs: 3 / Input size: 224x224 / Epochs: 5 / Dataset size: 1500
Batch size: 1 | Training time: 184 s | Gpu usage: 1462 MB
Batch size: 142 | Training time: 39 s | Gpu usage: 3591 MB
Batch size: 284 | Training time: 47 s | Gpu usage: 5629 MB
Batch size: 424 | Training time: 53 s | Gpu usage: 7523 MB
Batch size: 566 | Training time: 56 s | Gpu usage: 7770 MB
As u can see increasing batch size also increases total training time and this pattern is duplicated with other models.