Yes! In this case I’d recommend loading the encoder / decoder directly.
encoder = AutoModel.from_pretrained(...)
decoder = BertLMHeadModel(BertConfig())

enc_dec_model = EncoderDecoderModel(encoder=encoder, decoder=decoder)