That is weird. The model should work the same way whatever the batch size. It could be that your model is mixing inputs from different samples at some point in its forward method.
That said, if the model seems to be learning correctly then maybe there is a bug in pytorch, and in that case a minimal example that shows the error would be useful.