Ok, I do not know much about the EncoderDecoderModel so cannot comment on that. One thing I would like to understand is your setting for decoder_start_token_id. How come you decided to go with the pad token as opposed to a special token (e.g., <s> being a common one in translation). Is this how they train T5 to start with?