Your idea is using Adam to fast init training and turn to SGD at the end. I admit that is good idea and there is a paper called “Adaptive Gradient Methods with Dynamic Bound of Learning Rate. In Proc. of ICLR 2019. 30” has same idea.

As described in the paper, AdaBound is an optimizer that behaves like Adam at the beginning of training, and gradually transforms to SGD at the end

Please refer to this code: https://github.com/Luolc/AdaBound 44