JWarlock:

I must say returning nan gradient while computing std on all constant tensor is just so strange! The natural way is to return all 0 gradient, isn’t it? Maybe PyTorch should fix it.


I don’t think it’s natural to want the gradient of std at constant vectors, but yes, you could argue that picking 0 as a subgradient is better than NaN (which is likely artificially there from writing it as var(…)**0.5).