How to use Roberta as the Encoder and a randomly initialized TransformerDecoder as the Decoder?
Hello！
I want to use Roberta as an Encoder and a randomly initialized TransformerDecoder as a Decoder. I know EncoderDecoderModel can make any pretrained autoencoding model as the encoder and any pretrained autoregressive model as the decoder. But I just want to use 4 or 6 TransformerDecoder layers as my decoder. 
Maybe I should use torch.nn.TransformerDecoder to build my model. But is there any easier way or example when I use Huggingface Transformers?
Thank you!
@patrickvonplaten @NielsRogge
Hey @Captainr22, 

I would recommend doing the following:

- 1) Create a randomly intialized `RobertaModel` as your transformer decoder:
```python
from transformers import RobertaModel, RobertaConfig

decoder = RobertaModel(RobertaConfig(num_layers=4))
decoder.save_pretrained("./temp")
```

- 2) Now load an encoder / decoder model as follows:

```python
from transformers import EncoderDecoderModel

model = EncoderDecoderModel.from_encoder_decoder_pretrained("roberta-base", "./temp")
```

<|||||>Thank you for your recommendation！