Hi @neuralpat, I believe you are right that you can fuse BERT with GPT-2 checkpoints with EncoderDecoderModel although I suspect the performance may not be great given this table from the paper that this class is based on (look at the BERT2GPT row):
Screen Shot 2021-02-13 at 2.19.24 pm961×811 156 KB
So you might be better off trying to find a RoBERTa model in your target language, and using that as the encoder instead of BERT.
Now whether this will work better than fine-tuning BART or T5 is not obvious to me - as with most things in deep learning you probably have to determine it empirically