The loss looks indeed a bit fishy.
You don’t have to divide the loss by the batch size, since your criterion does compute an average of the batch loss. If your batch size is constant, this can’t explain your loss issue.
Could you post some more information regarding your experiment?
How high is your learning rate? Could you post your model architecture?
What kind of data do you have?
It the loss increasing in each epoch or just the beginning of training?