Hi @moscow25, thanks for sharing the AdaFactor info. I was wondering if constant LR of 1e-3 is working for small batch sizes, because in the paper they mentioned that the BS for fine-tuning was 128, it’s not possible to use 128 BS with single V100 for model >t5-base .