Artificial Intelligence, Technology, Conference, Iclr 2020, Neural Networks.
learning rate at each epoch given the current training loss. Figure: Training loss and normalized margin during training a CNNs, with and without bias on MNIST. The optimizer is SGD. (a) with a fixed learning rate 0.01, (b) with loss-based learning rate scheduler. source: Lyu, K. and Li, J. (2020).