Hi Frank!
Once again, thank you for your reply.
First of all, here is a zoom on the first graph around the drop at epoch 9 so that you can have a better view of the individual points (the 400th mini batch here is the batch 0 of epoch 9):

RollingZoom1170×876 221 KB




 KFrank:

This means (…) that the relevant p_{t+100} has to be lower than p_t about 80% or 90% of the time.


Obviously it’s just an example, but to me it seems that over the span of the drop, the most important part of the p_{t+100}s are indeed lower than the corresponding p_ts.



 KFrank:

Averaging over 100 such values should reduce the spread down by a factor of sqrt (100) = 10, so down to about 0.03.


Concerning the noise : I don’t know if it really helps, but I computed the std for the mini-batches 0 to 300 (as numbered in the above graph, using np.std), and the result is ~0.07. The std of the rolling loss over the same mini-batches is ~0.009.
The same computation for the mini batches 400 to 800 yields ~0.07 and ~0.005. So it seems that averaging over 100 values does indeed reduce the spread by a factor of ~10?



 KFrank:

Would it be possible to run 100 independent training runs (instead of 5), and average those runs to make a graph of the (averaged over runs) p_t s, but without any rolling-loss average?


Training 100 models would be quite long on my computer, but I can at least try to get more than 5. Here are some temporary results with 25 runs (more are training as I’m writing these lines, I’ll provide more graphs when I’ll have more data). If it’s not exactly the kind of graph you wanted, please tell me.

Average251170×876 58.3 KB


Drop251177×876 195 KB


Zoom2Epochs1177×876 102 KB




 KFrank:

Are these values approximately correct?


Yes they are. I’m using a batch-size of 32, and there are precisely 1562 batches per epoch (all of them being of size 32 because of the drop_last parameter in the train loader, otherwise the 1563rd batch would be of size 16).
I see your point concerning the fact that the network could have an advantage on recently presented samples. I will try using the RandomSampler as soon as my computer is finished with all the runs for the graph you previously asked for.
Thanks,
William