It is fairly common for a neural net to converge and then diverge. That is why many people monitor the validation loss and either reduce the learning rate or stop training when the validation loss stops going down.