Turns out I was calculating the losses wrong. After changing the calculation to
val_loss += loss.item() * len(names)

and
val_loss /= len(val_loader.dataset)

(The same for train loss)
I got the losses that are in accordance with what I think loss curves should look like.
Charts with new losses (with early stopping implemented):