GPT-2 FineTuning on Cloze/ ROC
Hi, wrote some code to finetune GPT2 on rocstories using the DoubleHeads model mirroring the GPT1 code. However, I'm only getting performance of 68% on the eval. Was wondering if anyone else had tried it and seen this drop in performance. Thanks
Hi rohuns, I was wondering what padding value have you used for the lm_labels, since the -1 specified in the docs doesn't work for me on GPT2LMHead model. See #577. <|||||>> Hi rohuns, I was wondering what padding value have you used for the lm_labels, since the -1 specified in the docs doesn't work for me on GPT2LMHead model. See #577.

I had just used -1, can take a look at your stack trace and respond on that chat<|||||>Also to close this issue it appears others also achieved similar performance on the MC task, more details on the thread issue #468 <|||||>> > Hi rohuns, I was wondering what padding value have you used for the lm_labels, since the -1 specified in the docs doesn't work for me on GPT2LMHead model. See #577.
> 
> I had just used -1, can take a look at your stack trace and respond on that chat

Yes, please do have a look. Here is a toy example with a hand-coded dataset to prove that the -1 throws an error. It looks like it's a library issue.
[gpt2_simplified.py.zip](https://github.com/huggingface/pytorch-pretrained-BERT/files/3144185/gpt2_simplified.py.zip)

Regards,
Adrian
