Worse performance of gpt2 than gpt
## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Hi, I just want to compare the performance of gpt and gpt2 as Language Model to assign Language modeling score. Like #473 , I implement my model as follows:
```
def gpt_score(text, model, tokenizer):
    input_ids = torch.tensor(tokenizer.encode(text)).unsqueeze(0)  # Batch size 1
    input_ids = input_ids.to('cuda')
    with torch.no_grad():
        outputs = model(input_ids, labels=input_ids)
    loss, logits = outputs[:2]
    sentence_prob = loss.item()
    return sentence_prob
a=['there is a book on the desk',
                'there is a rocket on the desk',
                        'he put an elephant into the fridge', 'he put an apple into the fridge']

tokenizer = OpenAIGPTTokenizer.from_pretrained('openai-gpt')
model = OpenAIGPTLMHeadModel.from_pretrained('openai-gpt')
model.to('cuda')
model.eval()

print([gpt_score(i,model,tokenizer) for i in a])

#config = GPT2Config.from_pretrained('gpt2')
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
model = GPT2LMHeadModel.from_pretrained('gpt2')
model.to('cuda')
model.eval()
print([gpt_score(i,model,tokenizer) for i in a])
```
And I get the following result:
```
[3.0594890117645264, 4.373698711395264, 5.336375713348389, 4.865700721740723]
[4.475168704986572, 4.266316890716553, 5.423445224761963, 4.562324523925781]
```
It seems that GPT get more sensible result than GPT2, but since gpt2 is literally gpt training with more data, how's that possible?

@Nealcly Could you try to use the `gpt2-medium` model? It has more layers :)<|||||>Also I'm not sure that your testing procedure is statistically representative :)<|||||>Any new updates on this issue? I am also facing the same question. <|||||>Based on the paper, only the largest model is called gpt2. The smallest model(117m) doesn't guarantee better performance than gpt.<|||||>how to finetune the language model with dataset and then get perplexity scores
<|||||>@anonymous297 please check the [documentation examples](https://huggingface.co/transformers/examples.html#language-model-fine-tuning), in which there's exactly what you're looking for.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
