Why do the optimizers require positive learning rate by default?
In some cases, the loss is maximized for one network and minimized for another network. It seems ideal to set one optimizer with positive learning rate and another negative learning rate instead of forwarding the input once more to compute (-loss).backward().