Can hidden states be passed instead of input_ids or inputs_embeds in Transformers OpenAI GPT2?
I am working on an encoder decoder model which uses a fine tuned RoBERTa as the encoder and GPT2 as the decoder. Before passing the encoder context to the decoder, I am mixing it with some context from a different domain. This mixing module is a simple NN. Hence, I now want to pass these transformed hidden states to the GPT2 decoder to do decoding, and I will train the decoder and the mixer only, not the encoder. How can I pass these transformed hidden states to the GPT2 decoder instead of the `input_ids` or `inputs_embeds`? The shape of my transformed hidden states is `(n_layers, batch_size, 1, hidden_size)` and I am currently using `batch_size=1`.  Any help will be appreciated. 
Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discuss.huggingface.co) instead?

Thanks!<|||||>Thank you! I have asked my [question](https://discuss.huggingface.co/t/can-hidden-states-be-passed-instead-of-input-ids-or-inputs-embeds-in-transformers-openai-gpt2/8073) on the forum.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.