Hello Abdeljalil!



 abdeljalil:

Executing the exact same code different times, most of the times I get that non-changing loss, sometimes it works and converges within a couple of epochs to 100% test accuracy as expected.
…
What can be the cause of this behavior?


This particular issue does not sound like a bug in your code,
although I suppose that would be possible.
More likely is that your training procedure is not robust for this
particular network / dataset.
(To clarify, is you loss literally unchanging, or is it just “stuck,” and
not making progress?  Have you tried letting your training run for a
long time to see if it can get off of its plateau?)
First, I would switch to SGD instead of using the Adam optimizer.
(I don’t have much experience with Adam, but the lore seems to
be that when it works, it can converge faster, but that it gets stuck
more easily.)
Second, I would try different learning rates, both significantly larger
and significantly smaller.  Depending on the nature of your plateau
or “getting stuck,” a larger or smaller learning rate can help address
the issue.  If this works, but is suboptimal (e.g., too slow), you could
then try using a learning-rate scheduler.
As for the difference between pytorch and what I assume is
keras / tensorflow, speaking from memory, scale their initial
random weights differently.
However, your training procedure should be robust with respect to
the general scale of you initial weights (and certainly your random
number seed).  So you could try changing the scale of your initial
weights, but even if that were to work, I would say it’s the wrong fix.
In short, I would suggest using SGD and trying various learning
rates.  If something appears to work, make sure it works with most
(all?) initial random number seeds.  If you get something working,
you can then back up and try to fine tune it to get it to run faster.
Good luck.
K. Frank