If anyone is facing the same issue, I was able to solve it using the MultiStepLR method.
In the training section i call the MultiStepLR at epoch 25 and 45 right after the optimizer.step() functions
the scheduler can be defined as
scheduler_G = MultiStepLR(optimizerG, milestones = [25, 45], gamma = gamma) scheduler_D = MultiStepLR(optimizerD, milestones = [25,45], gamma = gamma)
and it is called as
scheduler_D.step()
scheduler_G.step()