Hi Mr. Oraware!



 TheOraware:

" but this is easier to hand-wave away with more
“hard” samples in your training set or normalization vagaries, etc."


Some context:
As a general rule, you would expect (after some training) your training
loss to be smaller than your validation loss.  This is because, when
training on your training data, your model “learns” not only the general
features of the problem you are trying to solve, but also some specific
features of your training set.  That is, it “learns” (or perhaps “memorizes”)
that it should make certain specific predictions for certain specific
samples in your training set.  So your model performs better than it
“should” on these training samples it has “memorized” and therefore
produces a smaller training loss.  But, as you note, you are seeing
the opposite.
“Hard” samples:  Consider digit classification.  If sloppy enough a “1” and
a “7” can be hard to tell apart, as can be a “4” and a “9.”  Suppose that,
by happenstance, your training set has a lot of sloppy "1"s and "4"s, but
your evaluation set doesn’t.  When you make predictions on your training
set, you’re asking it to something harder – correctly recognize those
sloppy "1"s – than you’re asking it to do with your validation set.  So you
get a larger (worse) loss on your training set.
Normalization:  You normalize both your training set and validation set
using statistics computed from you training set.  (I think this is the right
thing to do.)  But your loss criterion – L1Loss scales with the size of
target (and predictions).  Let’s say that your training set has a “mean”
of 1.0 so that (in simplified terms) your normalization doesn’t actually
change anything, but, by happenstance, your validation set has a “mean”
of 0.9 – and this is not changed by normalizing it with the training-set
statistics.  Now, all else being equal, your validation-set loss will be
smaller by a factor of 0.9 than your training-set loss, merely because
the values given to your L1Loss criterion are, themselves, smaller by
a factor of 0.9.
One comment here:  From your graph, it appears that your training
loss starts out higher than your validation loss, even on the first batch
of each, which, if I understand your code, is before any optimizer steps
for your first training batch, but after one full epoch of training for your
first validation batch.  So this is not an apples-to-apples comparison.
In order for your training and validation losses to be comparable,
you want your training and validation set to have the same overall
characteristics.  Do you have a single larger dataset that you randomly
split into a training and validation set?  Or do your training and validation
data come from two separate sources?
Try taking a single dataset (for example, combine together your current
training and validation sets if they come from different sources) and
randomly split that single dataset into training and validation a handful
of times and run your training on those multiple random choices for
your training and validation data.  See if any systematic discrepancy
occurs across multiple choices of the dataset split.
Lastly, I don’t have good intuition about how the differing behavior of
your BatchNorm1d layers in train() and eval() modes might affect
your losses.
To explore this you could, after each epoch of training, put your model
in eval() mode (and, of course, not call optim.step()) and calculate
your training and validation loss (on batches of the same size).  Plot those
as a function of epoch and see if the discrepancy persists.
Best.
K. Frank