Continuing the discussion from Gradient Ascent and Gradient Modification/Modifying Optimizer instead of Grad_weight:
I’m working on a similar problem where I need to optimize the following loss function:
IMG_20200926_2229553106×4096 176 KB
Here w (omega) is model parameter and Lamdas are Lagrange Multipliers. I need to perform gradient descent wrt. omega and simultaneously gradient ascent wrt. lambda. lambda is not a model parameter and only included in the loss term.
Will your solution of updating lambda using gradient descent on -L work in this case? If it does then taking negative learning rate for lambdas in gradient descent should also be equivalent. And if it doesn’t then what should be the pytorch solution for this(without changing the optimizer source code)? Or should I need to creat a custom optimizer?