Yes I saw. It’s not exclusive to SGD. But the PR and the torch contrib implementation doesn’t mention this aspect of reproducibility issue. Many people have seen improvements as per the blog. I wonder how their LR scheduling is different from cyclical LR (in what ways is it better ?). Btw why are cyclical schedulers not used with transformers ?