The only reason why I put generate there is for debugging purposes because I noticed that the model produced non-sense in inference mode while achieving solid accuracies on the training and validation dataset. I wanted to make sure it’s not related to saving/loading issues or encoding/tokenization issues by using the very same model and input_ids. I also observed the outputs in training at early and late stages - generate keeps producing non-sense.
Note that I was able to successfully use the same code with the EncoderDecoderModel, I only had to switch the initialization of the tokenizer and the model.
I’m using the T5 pre-trained model as an init for a general sequence-to-sequence task, a very simple extraction task, still a toy problem, the sources are always of the form put x into y with x ; y being the targets. So the model should be able to reach near-perfect accuracies.