Solved by ptrblck in post #5 
              
              
                Your approach might work, if you are using a simple optimizer, e.g. SGD without momentum. 
When you are using momentum or an optimizer with running estimates, the reversed step might not work out of the box and you might need to look into the applied formula to check, how to revert it. 
Here is a sm…