I believe you are right, a higher batch size should give a better result. However, a higher batch size decreases the amount of optimization steps.
So try increasing the number of optimization steps / epochs by 50% and tell us how the results fair then.
Or do you possible have batch norm layers in your model? If you do, a batch size of 6 on 3 GPUs mean each GPU get 2 images -> not so nice for batch norm