Solved by ptrblck in post #6 
              
              
                Would you like to lower the learning rate to its minimum in each epoch and then restart from the base learning rate? 
If so, you could try the following code: 
model = nn.Linear(10, 2)
optimizer = optim.SGD(model.parameters(), lr=1.)
steps = 10
scheduler = optim.lr_scheduler.CosineAnnealingLR(optimi…