Extending Encoder Decoder to GPT-2
Adding GPT2 initialization for EncoderDecoder model as pointed out in the issue below.

> Currently, only Bert works as a decoder. We might add GPT2 in a couple of weeks. Note that no model has `cross-attention` layers if it is not already an encoder-decoder model (like Bart or T5) and in this case it does not make sense to use the encoder-decoder wrapper. The model is initialized with random weights for the cross attention layers which will have to be fine-tuned. I agree, that this should be made clearer in the documentation!

_Originally posted by @patrickvonplaten in https://github.com/huggingface/transformers/issues/4517#issuecomment-638058577_
It's on the roadmap :-) <|||||>Thank you! Look forward to it :)<|||||>Hi - I've actually been working on this myself the past couple days, should I submit a PR when finished? <|||||>That'd be great!<|||||>Will do - likely sometime this week.  <|||||>@djw1809 Any update on the PR? :)<|||||>@patrickvonplaten Hello Patrick, I am watching with much interest EncodeDecoder from transformers :) . Any updates on supporting GPT2 with EncodeDecoder ?<|||||>Got sidetracked with other research - coming back to it in several days,
working on my end, just need to play nice with the rest of the repo.

On Tue, Jul 7, 2020 at 3:32 PM Mihai Ilie <notifications@github.com> wrote:

> @patrickvonplaten <https://github.com/patrickvonplaten> Hello Patrick, I
> am watching with much interest EncodeDecoder from transformers :) . Any
> updates on supporting GPT2 with EncodeDecoder ?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/4961#issuecomment-655170674>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AG3PODZXYPBB33F4CBSNZLDR2OO7VANCNFSM4N4QTZQA>
> .
>


-- 
Dylan Weber, Research Assistant | PhD Candidate
School of Math and Statistical Sciences
WXLR642/BYENG593 Arizona State University
<|||||>@djw1809 - also feel free to already open a PR with unfinished code yet so that I can take a look early on and help you :-) <|||||>Working on it now. Also linking this PR: #4483<|||||>@patrickvonplaten  Hello Patrick. 
As I see from https://github.com/huggingface/transformers/commit/1d6e71e1167dea9e026391ec5a1a2d7ec33d22af current cross attention implimentation assume that encoder have same hidden size as GPT-2. I have encoder with hidden size 512 and want to combine it with GPT-2 medium with hidden size 1024. I have done it by Fairseq code and now want to do same by Huggingface. Could you update your solution to support any suitable encoder hidden space size?<|||||>Hey @Squire-tomsk, 

I see what you mean - this would mean to add a new config param for each model that has cross-attention...is this common practice? Would be great if you could open a new issue for that :-) <|||||>Done https://github.com/huggingface/transformers/issues/6645<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
