Why only use the hidden state of last token of last layer is used for predicting the next word?
I was trying to generate text and also reading the code to understand how it is working. 
I found that, after providing some text as context (first iteration), it goes through the transformer and the output of the transformer (`output[0] `of `GPT2Model` ) is for each token position there is a vector. To my understanding, these vectors are the context-aware representation of each token position. 
Now for generating the next word, the representation of the last token of last layers is being used.
This is the case for the first iteration.  
Then for each subsequent iteration, only the representation of the last predicted word is used to predict the next word. 

My question is, why the only the representation of the last word is used to predict the next word?
 
This raises another question, is the last token-position representation hold the context of the whole sequence (like LSTM)?   
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
