Seems that one has to use --evaluate_during_training in order for this to succeed. I was very sidetracked by the warning
 "FutureWarning: The 'evaluate_during_training'  argument is deprecated in favor of 'evaluation_strategy'
appearing in the run, and this pushed me to remove it, but its presence allows to print eval losses/metrics.