Hi William!



 WilliamWithering:

It worked ! With the replacement method there are no more drops.


To me, this is strong evidence that there isn’t some weird bug
in your code causing the epoch-boundary drops.
I’m very much believing the theory that the drops are caused
by the fact that the new batches near the start of a new epoch
can contain samples that were recently seen in batches near
the end of the previous epoch.


Concerning your 100 runs graph, I’ll stop at 50 (it takes too much time). There isn’t much difference with 25, just a little bit less noise and more defined drops.


No worries.  There’s nothing magic about 100 runs (or even 50).
As long as you have enough runs to be able to see through the
noise, that’s enough.


So it seems that the problem was coming from the DataLoader. I’d still like to know why, as I think that my first approach should have worked…
…
Is there any known issue with the DataLoader or the Sampler? Maybe a statistical bias or something like that?


I’m inclined to doubt that there is anything wrong with the
DataLoader (other than that it’s doing what it was designed
to do – which may or may not be what you want).
The DataLoader gives you an entire epoch of samples, one batch
at a time, without any repeats.  When you go on to a new epoch,
you sometimes will get repeats of recently seen samples.  You
would expect this to give the network an advantage, and lead to
a lower loss.  The only question is whether this effect would be
large enough to be significant and/or noticeable.  (I suppose
you could call this a “statistical bias” of a sort, but it is what the
DataLoader is supposed to do.)
(I think this theory could be tested, but at the cost of writing a
custom Sampler.)
If it’s easy, I would be interested in seeing a head-to-head
comparison of graphs (averaged over the same number of runs,
etc.), where the only difference is using the shuffled, no-repeat
DataLoader vs. the drawing-with-replacement RandomSampler.
Best.
K. Frank