Solved by albanD in post #6 
              
              
                It depends on your optimizer. 
If you don’t have momentum/accumulated terms, then you can simply set these gradients to 0 and your optimizer won’t change the values. 
If you have a fancy optimizer that will update the weights even for a 0 gradient, the simplest solution might be to save the original…