For HuggingFace GPT2DoubleHeadsModel, is there a way to directly provide a hidden state for an input? 
Hello,
Say I have two custom trained HuggingFace GPT2DoubleHeadsModels (Model 1 and 2).
 I want to take an hidden state of m-th layer of model 1, and use that hidden state as my input for model 2.
Is this possible with HuggingFace GPT2DoubleHeadsModels?

Some coding example would be a great help!

Thank you,

Hi! You have access to the models' internals so you could definitely rewrite the main loop function to handle such a use-case!

You can access the hidden layers as follows:

```py
blocks = model.transformer.h  # this is a list of "block"s
block[0]  # contains the MLP/LayerNorm/attention layers
```

I'd recommend taking a look at the `forward` method of the [`GPT2DoubleLMHeadModel`](https://github.com/huggingface/transformers/blob/master/transformers/modeling_gpt2.py#L642) to 
understand how the `transformer` is managed, and to take a look at the [loop inside the `GPT2Model`](https://github.com/huggingface/transformers/blob/master/transformers/modeling_gpt2.py#L451) which you will probably have to replicate a little to achieve what you want to do.<|||||>Hello, 
Thank you very much for your reply.
So if I am understanding this correctly, the part of the loop inside GPT2 model below is what I am supposed to use to generate output based on a given hidden state, am I correct?
```
outputs = block(hidden_states,
                            layer_past=layer_past,
                            attention_mask=attention_mask,
                            head_mask=head_mask[i])
```
Also, I would like to clarify what ‘block’ actually refers to in this code. Is ‘block’ same as GPT-2 ‘layer’? It’s not very clear to me what `block[1]`, `block[2]`, etc. represents.

Thank you again,<|||||>Yes.<|||||>Hello,

What exactly is the `block`? Does `block[0]` refer to the first gtp-2 layer, and `block[1]` refers to the second gtp-2 layer and so on? How can I utilize `block` to have access to the uppermost output layer of a gtp-2?

Thank you,<|||||>Yes, `block` refers to the GPT-2 layers. `block[0]` is the first layer, ... `block[-1]` is the last layer.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
