OK, it looks like I can now answer my own question. I can’t find any documentation of this, but it seems that the predict_with_generate flag WILL generation a text file with predictions, but not for each checkpoint – instead, it happens at the end, after all the epochs are complete.
The relevant code: transformers/run_seq2seq.py at f991daed185261085301d72c2cd634836df1044a · huggingface/transformers · GitHub 4
If anyone figures out a way to get run_seq2seq.py to generate these predictions at each checkpoint, my understanding was that this was the previous behavior, and it was certainly useful for me…