Artificial Intelligence, Technology, Conference, Iclr 2020, Neural Networks.
almost every neural network admits a chain rule (as long as the neural network is composed by definable pieces in an o-minimal structure, e.g., ReLU, sigmoid, LeakyReLU).” For the analysis on gradient flow, only local Lipschitz is required. While for analysis on gradient descent, C² smoothness is