Hi William!



 WilliamWithering:

Here is my problem :
I’ve been playing around with some neural networks lately,


Well, there’s your problem!  (Just kidding.)


I’m also plotting a rolling loss to smooth everything a bit, computed as follows :

And here is the kind of result that I get :
…
The steps which are appearing between every epochs shouldn’t be there I guess…


This does look odd.  I don’t see anything obviously wrong, but my
guess is that you have a gremlin hiding in your code somewhere,
and maybe it is in some part of the code you didn’t post.
But before I indulge in any idle speculation, let me point out that
there is something very fishy about your graph.
The rolling loss at time (batch iteration) t and t + 1 are related by
p'_{t+1} = p'_t + (p_{t+101} - p_t) / 100.  The drop in the
rolling loss in your graph at around epoch 15 is about 0.05.  Therefore
the difference between p_{t+101} and p_t would have to be about
5, that is, about twice the entire height of your graph!
(Even if your rolling loss is not averaged over five runs like your mean
loss (blue) is, you can’t realistically have a delta-p of 5 get averaged
down to something like the worst case of perhaps 0.3 that could be
hiding in the envelope of your blue graph.)
I could be misinterpreting your graph, or the numbers could be off,
but either way, I can’t really conjecture much at this point.
(As a side note, as written, your formula for your rolling loss sums
over 101 values of p_i, but divides by 100 – not that this is relevant
to the issue at hand.)


(PS : what’s the best way to integrate LaTeX equations in a post?)


My preference is not to integrate LaTex into this kind of forum.
Whether it’s some kind of embedded LaTeX (e.g., MathJax) or
an image generated from LaTeX, it’s harder to copy and paste
and to search on.  (But that’s just me …)
Good luck.
K. Frank