Solved by danitamm in post #2 
              
              
                Hello, 
I was wondering the same thing with respect to huggingface transformers’ scheduler. I found that one of this library’s examples addresses this in the scheduler constructor by dividing the “pre-accumulation” number of steps by gradient_accumulation_steps: 
t_total = len(train_dataloader) // a…