raharth:

Clipping gradients is not related to your output range I guess, it just prevents the gradients from exploding.


Yes, what I meant by them is, that you clip the output in the range of [-10,10] (not the gradients of the outputs). Although, clipping the output would in effect set the gradients to 0 for any output with |y| > 10 (like you said).



 raharth:

I’m not sure if I got you correctly, but what I meant was not just rescaling, but rescaling and using a sigmoid/tanh as output activation, then you have a constant gradient and do not have the issue of your values running “out of bounds”.


What I mean is, if you re-scale the outputs you can have issues where outliers will affect the rescaling. For example, if most outputs are near say -0.5, and a few outputs are near 1. Then the rescaling will be distorted by the outliers near 1. One way to circumvent that is to clip the outliers and then rescale. Having a bounded activation function will mitigate this, but it can still be an issue. It’s just something to keep in mind when rescaling data!