Hi Richard!



 Richard_S:

they implemented this in keras using
tf.keras.constraints.NonNeg()

So what is the most optimal way of implementing this in a multi layer NN in pytorch?


According to the keras documentation, Layer weight constraints 12:
“They are per-variable projection functions applied to the target
variable after each gradient update.”
So following along with what keras claims it does, you could try:
optimizer.step()
with torch.no_grad():
    self.classify.weight.copy_ (self.classify.weight.data.clamp(min=0))

to force the constraint after each optimizer step.
You would then hope that the training process causes the optimizer
to move the last layer’s bias into negative territory so that you get
predicted logits centered around zero, and therefore make “negative”
as well as “positive” predictions.
Such an approach would seem to be training the model with one hand
tied behind its back, but, in principle, it ought to be able to train the
biases to become negative.
Good luck.
K. Frank