dunefox:

For this you can just assume that the regularisation is correct.


I’m still unsure if your approach is right.
Note that you are subtracting a constant tensor with 1s at some positions from self.model.adapt.weight, while it seems your approach would rather subtract the values of self.weight at these positions from self.model.adapt.weight?



 dunefox:

How do I regularise with a custom regularisation scheme like this?


If you are convinced that the formula is right and I’m misunderstanding the use case, then the approach should work. You could just add the regularization loss to the final loss and call backward on it.