What does 'output of the embeddings' mean?
Hello,

According to Hugging Face Transformers documentation, (https://huggingface.co/transformers/model_doc/gpt2.html#gpt2doubleheadsmodel)
the transformer's output ```hidden_state``` is defined as the following:

```
hidden_states: (optional, returned when config.output_hidden_states=True)
list of torch.FloatTensor (one for the output of each layer + the output of the embeddings) 
of shape (batch_size, sequence_length, hidden_size): Hidden-states of the model 
at the output of each layer plus the initial embedding outputs.
```

I am a bit confused by the statement ```list of torch.FloatTensor (one for the output of each layer + the output of the embeddings) ```. Does the ```output of the embeddings``` at the end of the statement refer to 'output of the uppermost output layer'?

Thank you,

The output of the embeddings is the sum of the token embeddings + the segment embeddings + the position embeddings. This value is the value that will be fed to the first layer of the transformer.<|||||>@LysandreJik 

Hello,

Thank you very much for your reply.

So according to the Hugging Face Transformer documentation for the ```GPT2DoubleHeadsModel``` (under the 'output' section)

```
hidden_states: (optional, returned when config.output_hidden_states=True)
list of torch.FloatTensor (one for the output of each layer + the output of the embeddings) 
```
So in this case, would the first ```hidden_states``` tensor (index of 0) that is returned be the output of the embeddings, or would the very last ```hidden_states``` tensor that is returned be the output of the embeddings? 

I am confused about the order in which the ```hidden_states``` tensors are returned, because the documentation seem to indicate that the output of the embeddings is the last ```hidden_state``` tensor that is returned.

Thank you,<|||||>Indeed, the documentation might be misleading in that regard. The first value is the embedding output, every following value is the result of the preceding value being passed through an additional layer. I'll update the documentation shortly.<|||||>I remain confused by this and will be posting on the Disqus. 