I’m trying to reproduce your issue, but on my side, the best_metric is correct and decreasing. Could you check you are using the latest version of Transformers and post the way you are creating your TrainingArguments?