Problem with word prediction with GPT2
## ❓ Questions & Help
I'm trying to understand how to obtain the probability of specific word predictions, but I am getting bad results.  For example, according to the code below, the sequence "It seems that" is more likely followed by "ago" than by "we", which surely is not correct. What am I doing wrong?

```import sys
import torch
import numpy
from scipy.special import softmax
from pytorch_transformers import GPT2Config, GPT2Tokenizer, GPT2LMHeadModel

config = GPT2Config.from_pretrained('gpt2-medium')
tokenizer = GPT2Tokenizer.from_pretrained('gpt2-medium')
model = GPT2LMHeadModel(config)

item = "It seems that"
indexed_tokens = tokenizer.encode(item)
tokens_tensor = torch.tensor([indexed_tokens])

with torch.no_grad():
	predictions = model(tokens_tensor)
	results = predictions[0]    
	temp = results[0,-1,:]
	temp = temp.numpy()
	result = softmax(temp)	
	word_1 = tokenizer.encode('we')[0]
	word_2 = tokenizer.encode('ago')[0]
	print(result[word_1])
	print(result[word_2])
```

This outputs:
1.0500242e-05
5.1639265e-05
But it should be the other way around (see [here](https://books.google.com/ngrams/graph?content=seems+that+ago%2C+seems+that+we&year_start=1800&year_end=2000&corpus=15&smoothing=3&share=&direct_url=t1%3B%2Cseems%20that%20we%3B%2Cc0)).
Thanks in advance.
Hi, indeed it should be the other way around! I believe it's due to a misconception, you're initializing your model as follows:
```py
config = GPT2Config.from_pretrained('gpt2-medium')
model = GPT2LMHeadModel(config)
```
However, as noted in the [documentation](https://huggingface.co/transformers/main_classes/configuration.html#transformers.PretrainedConfig): _A configuration file can be loaded and saved to disk. Loading the configuration file and using this file to initialize a model does not load the model weights. It only affects the model’s configuration._

In order to initialize the model weights as well, you should do:
```py
config = GPT2Config.from_pretrained('gpt2-medium')
model = GPT2LMHeadModel.from_pretrained("gpt2-medium", config=config)
```
or since you're loading the pre-trained configuration of the same pre-trained model, you could simply do:
```py
model = GPT2LMHeadModel.from_pretrained("gpt2-medium")
```
which already loads this configuration file.

Once you have done this change you should get as output:

```
1.248501e-06
3.727657e-09
```

which is more accurate :)<|||||>Yes! Thank you! I was looking for an error on the completely wrong place (must have re-written the latter part of the code about 5 different ways).