ptrblck:

If you are convinced that the formula is right and I’m misunderstanding the use case, then the approach should work.


It is, I just explained it badly. 



 ptrblck:

You could just add the regularization loss to the final loss and call backward on it.


That’s what I’m doing; at least now I know that I’m not misunderstanding PyTorch itself in this case.
Thank you!