Hi K. Frank !
First of all, thank you for your reply 



 KFrank:

The rolling loss at time (batch iteration) t and t + 1 are related by p'_{t+1} = p'_t + (p_{t+101} - p_t) /100 . The drop in the rolling loss in your graph at around epoch 15 is about 0.05. Therefore the difference between p_{t+101} and p_t would have to be about 5, that is, about twice the entire height of your graph!


Your observation is really interesting, and I thought it would be the solution to my problem. However, I zoomed in on the graph, and the drop doesn’t occur exactly between p_t and p_{t+1}, but over the span of a hundred of batch iterations (I’ve got ~46K points in the entire graph, hence the impression of a sudden drop).
Here is a zoom on the drop at the 15th epoch :

Rolling_loss%20drop1177×876 18.6 KB

I don’t think that the rolling loss is the problem, because for me it seems that the drops are visible even in the mean loss (from the 6th epoch to the 14th for example). Just in case, this is how I compute it (train_bn being my list of averaged losses) :
rolling = [np.mean(train_bn[i:i+100]) for i in range(len(train_bn) - 100)]



 KFrank:

My guess is that you have a gremlin hiding in your code somewhere, and maybe it is in some part of the code you didn’t post.


Concerning the part of the code I didn’t post, here it is : https://pastebin.com/RFNLuLQG 5
And just in case, here is how I create the graph : https://pastebin.com/zmT5efJq 1
William