GPT-2 language model: multiplying decoder-transformer output with token embedding or another weight matrix
I was reading the code of GPT2 language model. The transformation of hidden states to the probability distribution over the vocabulary has done in the following line:

`lm_logits = self.lm_head(hidden_states)`
Here,

`self.lm_head = nn.Linear(config.n_embd, config.vocab_size, bias=False)`
However, In the original paper, they suggested multiplying hidden states with the token embedding matrix whereas huggingface implementation used another matrix.

Is there any advantage of this? Am I missing something?
Hi, the input embeddings are tied to the output embeddings -> The `lm_head` attribute essentially shares its weights with the embedding layer. Passing the output of the transformer through that layer is the same as multiplying this output (the hidden states) with the token embedding matrix.<|||||>@LysandreJik Where is the code that performs this weight tying?<|||||>I think I found it:

```
        output_embeddings = self.get_output_embeddings()
        if output_embeddings is not None:
            self._tie_or_clone_weights(output_embeddings, self.get_input_embeddings())

```
in `PreTrainedModel.tie_weights()` in `modeling_utils.py`.