I’m using gradient accumulation and torch.optim.lr_scheduler.CyclicLR.
Is there a special thing to consider when using gradient accumulation in this case?
scheduler.step() should come after optimizer.step(). However because of gradient accumulation, this may not be the case.
Thank you