AlphaBetaGamma96:

I think with clipping the gradient would only be zero for outputs out of the [-10,10] range


Clipping gradients is not related to your output range I guess, it just prevents the gradients from exploding.
Clipping the output, will as you said mess with the gradients outside of the range you have defined, but would work perfectly fine within them. Though in my experience this will cause trouble especially with REINFORCE, but potentially with all algorithms.



 AlphaBetaGamma96:

can have issues with outliers force most outputs near the extremes (i.e +/- 10)


I’m not sure if I got you correctly, but what I meant was not just rescaling, but rescaling and using a sigmoid/tanh as output activation, then you have a constant gradient and do not have the issue of your values running “out of bounds”. Should have added that to the original post (in fact I will now). Or did I get you wrong?