I found a solution which at least helps a little:
When using EncoderDecoderModel(), it is possible to set gradient checkpointing at least on the encoder part:
model.encoder.config.gradient_checkpointing = True