rwightman:

As someone else mentioned, the metric for plateau LR scheduler should be from validation, not train, regardless of whether it is an accuracy, loss, or other metric. Defeats the point otherwise as train loss can easily continue to fall while validation loss starts increasing once you overfit.


What a useful comment, thank you very much! Indeed I misused the scheduler. I’ll refine the learning procedure and see if it helps!


Differences in image processing (even interpolation), normalization, optimizer params or implementation (momentums, epsilons, etc), layer params or impl (epsilons, momentums), can all have an impact.


I understand that, I tried to pay attention to all the details.