@mrich This can happen with any NLL based loss optimizer.


You can introduce smart learning_rate reduction when you find that the loss is not reducing for 2 to 3 epochs, otherwise the graph can pick up some incorrect gradient and move in that direction


You can reduce the batch_size to 16 or 32 as this allows the model to train in a more generic direction rather that specific after every epoch


You can switch over from advanced optimizers like Adam etc to SGD after a certain reduction of loss so that the optimizer efficiently manoeuvres the gradient drops


This exercise is more of an art than science so you will only get tips. Good luck