Hi Richard!



 Richard_S:

    criterion = nn.BCELoss()
    optimizer = optim.Adam(model.parameters(), lr=0.001)



From your previous posts it appears that you have a multi-label,
multi-class problem.  For this, BCELoss is reasonable.  For
improved numerical stability, however, you should prefer using
BCEWithLogitsLoss and remove the sigmoid() call from your
forward() function.
Would it be practical for you to experiment with different optimizers,
or is your model too slow and expensive to play around with?
I would recommend starting with plain-vanilla SGD, if only to get a
baseline for comparison.
Both SGD with momentum and Adam “remember” things from
previous steps, so the fact that you will be munging your weights
after the optimization step might confuse this process.  It’s not that
you shouldn’t try SGD with momentum or Adam – they might work
fine – but be on the lookout for potential issues and use plain-vanilla
SGD as a sanity check.
In general, weight decay is worthwhile, so you should probably turn
it on (but run a baseline without it).
Your BCEWithLogitsLoss (or BCELoss) loss function should cause
your model to train to make some “negative” predictions (assuming
that your training data is sensible), but your weight-munging scheme
is, at best, going to make your training process more difficult.  So you
might have to carry out training runs that are longer than normal.
(Also, experiment with your learning rate.)
Lastly, if your training data is *unbalanced," that is, some given class
has many more “negative” samples than “positive,” you should also
consider using BCEWithLogitsLoss's pos_weight argument to
account for this.
Good luck.
K. Frank