I train a seq2seq model using SGD as the optimizer with momentum=.9
At the beginning the loss reduces as expecting, then the loss increase suddenly even over the initial loss.
BTW: how to upload the image? Does it depend on the user level? The plot of loss may show the result.