Can we directly replace gpt2LMHeadModel with BertLMHeadModel to see bert's performance? #7
I have a code for gpt2LMHeadModel which runs well, and I want to test my code on BertLMHeadModel. 
But when I directly replace gpt2LMHeadModel with BertLMHeadModel and replace gpt2Tokenizer with BertTokenizer, the ppl remains at 1 (the BertLMHeadModel predicts perfectly the same with labels)

So can anyone help me, is there any difference of the input format between gpt2LMHeadModelwith and BertLMHeadModel? 

Thanks so much!
Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discuss.huggingface.co) instead?

Thanks!<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.