Thanks for the reply @sgugger . Since loss may not be directly related to certain metrics, I was hoping to see where the metrics trend starts overfitting and make decisions based on that. Do you think this is a valid use case to think about adding this to the API in the future?