I am trying to implement a particular learning rate decay on the Adam optimizer with each training step (global step) according to the function below:
from math import pow


init_learning_rate = 1e-3
min_learning_rate = 1e-5
decay_rate = 0.9999

lr = ((init_learning_rate - min_learning_rate) *
      pow(decay_rate, global_step) +
      min_learning_rate)

optimizer = torch.optim.Adam(model.parameters(), lr=lr)

Would I have to redefine the optimizer at every time step?