You chose the batch size based on your data, model, and hardware.
If you are using BatchNorm layers in your model, your batch size should not be smaller than ~64, since then you could observe a degrading performance, because the running stats could be noisy.
This of course also depends on the data you are using.
Another limitation could be the hardware. If you have a large model and would like to train it on the GPU, you could run into out of memory errors with a large batch size.
Usually you would use some mini batches with a batch size > 1 and < size of your dataset.
The NaN in your loss might come from another issue in your code. Could you check, that no NaNs are in your data?