Hi,
@patrickvonplaten has an example of this here: patrickvonplaten/bert2gpt2-cnn_dailymail-fp16 · Hugging Face 5
Basically, you need to adapt the Trainer a bit in order for it to work with the EncoderDecoderModel framework. However, we are planning to improve the encoder decoder framework (as we’ve recently also added SpeechEncoderDecoderModel and VisionEncoderDecoderModel 1) for it to work with the Trainer by default.