I found an initial solution to the problem, which is re-scaling the gradients of each input sample so that the norm of the smallest sample is one. However, to keep using a regular optimizer, I need to figure out how that effects the momentum of the optimizer, etc. At the moment, I am just updating the input samples by adding them their re-scaled gradients. I am looking forward for any suggestions from more experienced.