This is interesting. The values in the batches are not unique in spite of  using replacement = False.
As far as the loss is concerned, This could be down to a couple of problems. Try the following out

Try out different learning rates (smaller than the one you are currently using).
Remove all regularization and momentum until the loss starts decreasing.
Check the inputs right before it goes into the model (detach and plot it). Check correspondance with labels.
You may also be updating the gradients way too many times as a consequence of a small batch size. As the targets are still not unique, you may as well keep a larger batch.