Why is the weight of linear layer tied to the input embeddings in OpenAIGPTLMHeadModel?
## ❓ Questions & Help
Yes the original GPT paper also uses same `W_e` as both token embedding matrix and linear weight, and seems that many succeeding models like GPT-2, XLNet also use the same matrix. In my perspective, the token embedding matrix and the weight in linear layer have nothing related (though they have both the same shape). Could you please explain a bit of that?

The token embedding matrix and the linear layer of the **language modeling head** are indeed tied. The embedding matrix is used to map the vocabulary to vectors of last dimension `hidden_size`. 

The linear layer is used to do the exact same thing, just the other way around -> mapping the model output of last dimension `hidden_size` to the vocabulary, so that the output may be converted into vocabulary tokens.<|||||>First of all, thanks for the replay!

I know that the last linear layer is to mapping `hidden_state` with the size `hidden_size` to the size of the vocabulary, but the linear layer does not need to output concrete tokens, right? It just needs to output a group of probabilities (with the size of vocabulary) with softmax, and these probabilities seem to have nothing to do with the token embedding matrix?

I have read some other papers, like the CBOW model in word2vec, which uses a linear layer with separate parameters before softmax to train the language model. As a result, the way that GPT does makes me feel confused.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
