Hello everyone!
I was thinking: is there a way to make a backward optimization step?
Say the learning rate was too high, or I have not understood what momentum is and but a nonsense value. But after having performed the optimization, I re-evaluate the loss function and decide that I’m not happy with the results.
So I want to make a step backwards, change parameters and try again, so that I would have a better optimization trajectory. Is it possible to do something like this?
For now my code is pretty basic, with the built-in ADAM:

outputs = model(images)
loss = lFun(input=outputs, target=targets)
optimizer.zero_grad()
loss.backward()
optimizer.step()

In other words is there a way of doing optimizer.backward_step()?