1 . --do_eval shouldn’t be mandatory. If we set --evaluate_during_training
without --do_eval then we see this issue. The bug is here 2 in finetune_trainer.py, it only loads eval_dataset when --do_eval is set, it should also check for
--evaluate_during_training or the new evaluation_strategy argument.
@sgugger I think this needs to be handled in every example/ script, WDYT ?
4 . Progress bars in Trainer are bit different than PL, it creates one progress bar for each epoch and each eval loop (eval pbar is created for each device when on muli-gpu or TPU ). And another one to track epochs.
5 . I’m also confused about why log_history is saved in every ckpt, @sgugger may know why.