It is not unusual to see a slight increase in training loss once the model converges.  However, something looks off here.  I would suggest printing accuracy as well to see how the correlation looks, as well as lowering the learning rate.