If I may ask a follow-up question: what is the heuristic for picking the number of warmup steps? Is it the first 6% of steps that one sees in the literature (e.g. the RoBERTa paper)?
The reason I ask is that I want to run some experiments on a subset of SQuAD and am wondering how I should scale-down the warmup_steps argument accordingly