I used Pytorch profiler to profile training and tensorboard to make it looking cleaner. I have tested densenet121 with batch_size 20 and 60. Here are some graphs that profiler has created.

densenet1211920×1080 163 KB

And as you can see, for batch_size=20 GPU usage was around 50% and average step time was almost two times faster than average step time for batch_size=60, even though the GPU consumption was lower, which confuses me.
Logs shows that kernel (which as far as I understand is GPU)  takes the biggest part of time and I don’t know whats the problem. If its not I/O that take time, neither data augmentation, why its still remains slower for bigger batches?