I suspect that we have a bug here: https://github.com/huggingface/transformers/blob/master/examples/seq2seq/finetune.py#L104-L106 4
We set decoder_start_token_id when the model is BART, but nothing is set for a T5 model.
Thoughts @sshleifer @valhalla?