Visiting from your comment on github!
I am just getting back to this project (finetuning t5) this weekend. I was planning to do similar investigations to what you’ve been doing here, since I also suspected something might be wrong with tokenizer / EOS/ masking.
I haven’t run any new experiments yet, but I still don’t quite understand why the model was struggling to learn to output EOS tokens when doing beam search generation (with a high max_len).
Also, per the comment to adjust n_train: I just want to confirm that setting n_train=-1 should have no effect? Or am I missing something.