Hello William!



 WilliamWithering:

I don’t think that the rolling loss is the problem, because for me it seems that the drops are visible even in the mean loss (from the 6th epoch to the 14th for example). Just in case, this is how I compute it (train_bn being my list of averaged losses) :


Something still seems inconsistent with your graphs.
Let me explain what my concerns are.  I can’t really back this
up with numbers because I can’t see the details of the blue
data in your first graph – it’s all blurred together.  All I can
really see is the envelope of the blue data.
So two concerns:
In your second graph – the rolling-average loss where I can see
the individual points – over the range of about 100 batches where
the rolling loss decreases nearly steadily, it looks like the next
point is lower than the previous about 80% or 90% of the time.
This means (from my discussion in my first post) that the relevant
p_{t+100} has to be lower than p_t about 80% or 90% of the
time.  But the blue envelope in your first graph makes it look to me
like the p_ts are very noisy, jumping back and forth.  I just can’t
imagine a realistic correlation that would cause such noisy data
to systematically satisfy p_{t+100} < p_t (over the relevant
range).
The second also has to do with the noise in the p_t that I deduce
from the blue envelope.  My understanding is that the blue data
in your first graph comes from averaging over five runs.  But let’s
say, just to make the point more extreme, that it’s a single run.
The spread in the data (which should be proportional to the
standard deviation) is about 0.3.  Averaging over 100 such values
should reduce the spread down by a factor of sqrt (100) = 10, so
down to about 0.03.
If I look at the “noise” wiggles in the orange curve in your first
graph – that is, not the sawtooth shape that tracks the epochs,
but rather the short-time fluctuations on top of the sawtooth –
(and, similarly, at the point-to-point, non-trend fluctuations in
your second graph), the noise looks rather smaller than 0.03.
That is, your rolling loss looks less noisy I would expect from
what the blue envelope seems to be telling me about the noise
in the unaveraged p_ts.
Again, it’s hard for me to make a clear-cut case about this, since I
can’t see the specific p_t values inside the blue envelope, and it’s
hard to prove the negative that there aren’t any weird correlations
that would cause the noise to average away unexpectedly rapidly.
If we take your rolling-loss graph at face value, it suggests that
averaging over 100 p_t values averages away enough noise
to make the effect clearly visible.  Would it be possible to run 100
independent training runs (instead of 5), and average those runs
to make a graph of the (averaged over runs) p_ts, but without
any rolling-loss average?
The rolling-loss average isn’t an illegitimate way to try to dig
some signal out of the noise, but it does complicate the analysis
somewhat (and the results do look fishy to me).  It’s not that I don’t
see the effect in the blue envelope – it’s just that rolling-loss graph
noise doesn’t agree with the simplest batch-to-batch fluctuations
suggested by the blue envelope.
If you do 100 runs, could you plot not only the whole (averaged)
training run, but also a couple of epochs and epoch transition
regions where we can see individual points in the graph?


rolling = [np.mean(train_bn[i:i+100]) for i in range(len(train_bn) - 100)]


As an aside, this code for the rolling average looks correct to
me (with the quibble that in your training loop you build the list
train_losses, but here you calculate the rolling-loss from
something called train_bn).
Also, based on assuming that you use the 50,000-sample CIFAR-10
training dataset for your training, I estimate that you batch size is
about 32, and that you have about 1,500 batches per epoch.  Are
these values approximately correct?
Now for some idle speculation:  It’s does look like something is
happening at epoch boundaries, but I can’t think of a straightforward
explanation that fits.
It appears that you use a shuffling dataloader to create your
batches, so this explanation doesn’t apply.  But suppose you didn’t
shuffle, and you simply generated batches in the original order
of the CIFAR data.  (I don’t know what that order is, by the way.)
If the CIFAR had easier samples towards the beginning, and
harder samples toward the end (perhaps ordered by class, and
easier classes happened to come first), we could easily see your
effect.
But you do shuffle.  Now, within an epoch, all the batches contain
different samples, so as you train, you don’t get the benefit of
seeing again a sample you’ve already trained on.  But across
an epoch boundary, the first batches you see could happen to
include samples that were seen recently in the last few batches
of the previous epoch.  This could give your network the advantage
of being asked to classify a sample that it had recent trained on
(from a batch in the previous epoch), thereby having an easier
task and a lower a loss.
However, with a batch size of 32, I just don’t see this happening
enough to produce a dramatic (or even noticeable) effect.
Nonetheless, if this were the case, you could do the following
test.  If you created batches by drawing samples with
replacement (@ptrblck – is there a pre-packaged sampling-
with-replacement dataloader, or a simple strategy to write one?)
then there would be no actual epoch boundaries – an “epoch”
would be a number of training samples equal to the size of
your training set, but some samples would occur (randomly)
more than once in an “epoch,” and some not at all.
If the sawtooth-loss structure persisted with this test, it would
indicate clearly a weird bug somewhere.
Again, idle speculation.  There is something going on at your
epoch boundaries, but it’s hard to guess what it might be with
the data partially hidden.
Good luck!
K. Frank