hmm that is odd indeed. the error seems to be coming from NotebookProgressCallback which expects metrics to have an eval_loss field: transformers/notebook.py at 02f7c2fe66cf3ef11402adc3d9d8a3ddd189c717 · huggingface/transformers · GitHub 11
as a dirty hack, what happens if you do the following before the on_evaluate step is called:
eval_output.metrics["eval_loss"] = "No log"
self.control = self.callback_handler.on_evaluate(self.args, self.state, self.control, eval_output.metrics)
self.control = self.callback_handler.on_evaluate(self.args, self.state, self.control, train_output.metrics)

this ensures the eval metrics have an entry for eval_loss before the callback is called. if this hack works, it might be a bug in the callback that we should fix