Solved by Seo in post #4 
              
              
                The warmup technique is used during the first phase of the training. It consists in small learning rate and a scheduler that increases it everytime until a certain number of steps (warmup steps). So you need 2 different scheduler, as you can see from the code you posted, the warmup scheduler seems t…