The Adam weight update is defined as
w_{t+1} = w_t - \eta \left( \frac{1}{\sqrt{v_t} + \epsilon} \right) m_t
(see here for rendered latex)
I would like to plot the “effective learning rate” of my optimizer during training, which requires knowing the moment estimates v_t and m_t. Does anyone know how I can extract these from an Adam optimizer?