hey @yansoares, you could try using the EncoderDecoderModel  (docs 51) to create a BERT2GPT model. I’m not aware of any tutorials using this class, but the original paper (link 55) is quite well written and should give you a good idea on what needs to be done.