It sounds a little cumbersome for something that should be straight forward.
I have the feeling that in the TF version https://www.tensorflow.org/api_docs/python/tf/contrib/opt/LazyAdamOptimizer 19
this is completely embedded in the lower level LazyAdam implementation.
Am I correct ?