Exactly!
I see the the optimizer as a blindfolded person walking randomly in the mountains and trying to get to the lowest point. And I would like sometimes to tell him to step back because he is not going in the right direction.
Seems I will have to create my own gradient descent function to see if something comes out of this idea.
Thanks!