The larger your batch size, the smaller on average your gradients. This also means that smaller batch size show a bit more erratic convergence.
Try lowering the learning rate for your smaller batch size.