Hello William!



 WilliamWithering:

First of all, here is a zoom on the first graph around the drop at epoch 9 so that you can have a better view of the individual points (the 400th mini batch here is the batch 0 of epoch 9):
…
Training 100 models would be quite long on my computer, but I can at least try to get more than 5. Here are some temporary results with 25 runs (more are training as I’m writing these lines, I’ll provide more graphs when I’ll have more data). If it’s not exactly the kind of graph you wanted, please tell me.


Thanks for the zoomed-in and 25-run graphs.


To my eye, they further demonstrate the epoch-boundary-drop
you showed in your first post.


Looking at these, I no longer see evidence for any discrepancy
between the rolling-average and single-batch results.  (My intuition
about how they should be related was clearly off.)


Don’t kill yourself trying to get 100 runs if it’s too slow and painful.
It would be nice, but not necessarily worth it if it’s a lot of trouble.


I see your point concerning the fact that the network could have an advantage on recently presented samples. I will try using the RandomSampler as soon as my computer is finished with all the runs for the graph you previously asked for.


There is clearly something going on at the epoch boundaries.
Unless there’s just a bug somewhere, the only thing that happens
at the epoch boundaries is the change in the character of the
batches (relative to recent batches in the previous epoch).
I’m leaning towards believing my story about the new epoch having
some samples be recent repeats.  (It’s really all I can think of.)  I’m
somewhat surprised that the effect could be so large, but I don’t
really have any intuition about the CIFAR dataset or the VGG16
network.
I do believe that if you create batches with random drawing with
replacement, there should be no change in batch character at
the epoch boundaries.  (In fact, there won’t really be any epoch
boundaries except those from “arbitrarily” counting the number
of batches.)  So the effect should vanish entirely.
If it doesn’t, I would say look for a bug somewhere.  But you have
a very simple nested loop where you loop over epochs and then
loop over batches, so I just don’t see where an epoch-boundary
bug could sneak in …
Anyway, I think your code is sound (in this respect – I’m not saying
there couldn’t be other little nits here and there …).  I think the test
with batch-sampling-with-replacement is the thing to do.
Good luck!
K. Frank