size mismatch for lm_head.decoder.weight
Hi i'm new to this,

first i started a finetune job
```
export ROC_STORIES_DIR=roc/

python run_openai_gpt.py \
  --model_name openai-gpt \
  --do_train \
  --do_eval \
  --train_dataset $ROC_STORIES_DIR/cloze_test_val__spring2016\ -\ cloze_test_ALL_val.csv \
  --eval_dataset $ROC_STORIES_DIR/cloze_test_test__spring2016\ -\ cloze_test_ALL_test.csv \
  --output_dir ../roc_gpt \
  --train_batch_size 16 \
```

i have the following files in the output folder
```
config.json  eval_results.txt  merges.txt  pytorch_model.bin  special_tokens.txt  vocab.json
```

however when i run 
```
python run_gpt2.py --model_name_or_path=../roc_gpt/
```

i get this error

```
(env) [wwilson@b-user-wwilson-m ~/persistent-disk/notebooks/pytorch-pretrained-BERT/examples]$ python run_gpt2.py --model_name_or_path=../roc_gpt/
Namespace(batch_size=-1, length=-1, model_name_or_path='../roc_gpt/', nsamples=1, seed=0, temperature=1.0, top_k=0, unconditional=False)
05/07/2019 09:58:24 - INFO - pytorch_pretrained_bert.tokenization_gpt2 -   loading special tokens file ../roc_gpt/special_tokens.txt
05/07/2019 09:58:24 - INFO - pytorch_pretrained_bert.tokenization_gpt2 -   loading vocabulary file ../roc_gpt/vocab.json
05/07/2019 09:58:24 - INFO - pytorch_pretrained_bert.tokenization_gpt2 -   loading merges file ../roc_gpt/merges.txt
05/07/2019 09:58:24 - INFO - pytorch_pretrained_bert.tokenization_gpt2 -   Special tokens {'_start_': 40478, '_delimiter_': 40479, '_classify_': 40480}
05/07/2019 09:58:24 - INFO - pytorch_pretrained_bert.modeling_gpt2 -   loading weights file ../roc_gpt/pytorch_model.bin
05/07/2019 09:58:24 - INFO - pytorch_pretrained_bert.modeling_gpt2 -   loading configuration file ../roc_gpt/config.json
05/07/2019 09:58:24 - INFO - pytorch_pretrained_bert.modeling_gpt2 -   Model config {
  "afn": "gelu",
  "attn_pdrop": 0.1,
  "embd_pdrop": 0.1,
  "initializer_range": 0.02,
  "layer_norm_epsilon": 1e-05,
  "n_ctx": 512,
  "n_embd": 768,
  "n_head": 12,
  "n_layer": 12,
  "n_positions": 512,
  "n_special": 3,
  "resid_pdrop": 0.1,
  "vocab_size": 40478
}

05/07/2019 09:58:26 - INFO - pytorch_pretrained_bert.modeling_gpt2 -   Weights of GPT2LMHeadModel not initialized from pretrained model: ['transformer.wte.weight', 'transformer.wpe.weight', 'transformer.ln_f.weight', 'transformer.ln_f.bias']
05/07/2019 09:58:26 - INFO - pytorch_pretrained_bert.modeling_gpt2 -   Weights from pretrained model not used in GPT2LMHeadModel: ['multiple_choice_head.linear.weight', 'multiple_choice_head.linear.bias', 'transformer.tokens_embed.weight', 'transformer.positions_embed.weight']
Traceback (most recent call last):
  File "run_gpt2.py", line 129, in <module>
    run_model()
  File "run_gpt2.py", line 77, in run_model
    model = GPT2LMHeadModel.from_pretrained(args.model_name_or_path)
  File "/mnt/notebooks/notebooks/pytorch-pretrained-BERT/env/lib/python3.6/site-packages/pytorch_pretrained_bert/modeling_gpt2.py", line 475, in from_pretrained
    "Error(s) in loading state_dict for {}:\n\t{}".format(model.__class__.__name__, "\n\t".join(error_msgs))
RuntimeError: Error(s) in loading state_dict for GPT2LMHeadModel:
        size mismatch for lm_head.decoder.weight: copying a param with shape torch.Size([40481, 768]) from checkpoint, the shape in current model is torch.Size([40478, 768]).
```

I'm guessing the first script is for gpt and the second one is for gpt2?
should i adjust the 2nd script to load the same classes as the trainer one to use the fine tuned gpt2 model?

thanks
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>```Traceback (most recent call last):
  File "question-generation/interact.py", line 238, in <module>
    run()
  File "question-generation/interact.py", line 144, in run
    model = GPT2LMHeadModel.from_pretrained(args.model_checkpoint)
  File "/usr/local/lib/python3.6/dist-packages/pytorch_pretrained_bert/modeling_gpt2.py", line 475, in from_pretrained
    "Error(s) in loading state_dict for {}:\n\t{}".format(model.__class__.__name__, "\n\t".join(error_msgs))
RuntimeError: Error(s) in loading state_dict for GPT2LMHeadModel:
	size mismatch for transformer.wte.weight: copying a param with shape torch.Size([50265, 768]) from checkpoint, the shape in current model is torch.Size([50257, 768]).
	size mismatch for lm_head.decoder.weight: copying a param with shape torch.Size([50265, 768]) from checkpoint, the shape in current model is torch.Size([50257, 768]).
```
I am having a similar issue? Any idea how to solve this? @Wingie <|||||>+1<|||||>+1<|||||>+1
<|||||>> ```
>   File "question-generation/interact.py", line 238, in <module>
>     run()
>   File "question-generation/interact.py", line 144, in run
>     model = GPT2LMHeadModel.from_pretrained(args.model_checkpoint)
>   File "/usr/local/lib/python3.6/dist-packages/pytorch_pretrained_bert/modeling_gpt2.py", line 475, in from_pretrained
>     "Error(s) in loading state_dict for {}:\n\t{}".format(model.__class__.__name__, "\n\t".join(error_msgs))
> RuntimeError: Error(s) in loading state_dict for GPT2LMHeadModel:
> 	size mismatch for transformer.wte.weight: copying a param with shape torch.Size([50265, 768]) from checkpoint, the shape in current model is torch.Size([50257, 768]).
> 	size mismatch for lm_head.decoder.weight: copying a param with shape torch.Size([50265, 768]) from checkpoint, the shape in current model is torch.Size([50257, 768]).
> ```
> 
> I am having a similar issue? Any idea how to solve this? @Wingie

Were you able to fix this? I am having this issue, with the same sizes as well.<|||||>same here!
