tom:

I thought Klambauer et al, Self-Normalizing Neural Networks  had the elaborate insights for this including gradients.


I understood it as: even after updating the weights, they will stay in the domain of the contraction mapping so that the fixed point does not change.