Hi @SarthakJain
Thanks for your input! I’m already doing linear learning rate warmup and cosine annealing to decrease my learning rate.
What I’m worried about is that my dirty data of 5% produces much more than 5% of the loss, hence affecting the model loss-landscape/trajectory more than I’d like (even if the learning rate is low).