Why GPT2LMHeadModel's loss mismatches accuracy?
# ❓ Questions & Help
Hello,
I'm pretraining GPT2LMHeadModel on my own data from scratch, however I find that loss decreases sharply, and the evaluation doesn't match the loss anymore.

## Details
I used two different ways to evaluate the LM model:
1.input [1,2,3...] label [2,3,4...] this way will compare the output of whole sentence with label once
2.input [1] label [2]
   input [1,2] label [2,3] but we compare only the output of 2's position and label 3
   input [1,2,3] label [2,3,4] but we compare only the output of 3's position and label 4
   and so on (each time count only last position's accuracy)
   finally sum up them

In my opinion, these two ways should give the same result, however when loss decreases sharply, these two results don't match anymore. This happens only in the case that learning rate is high, like 0.001 or even bigger.

Could you help me explain how it happens?
![loss](https://user-images.githubusercontent.com/62126666/80560370-4089b600-8a13-11ea-9b7f-76ff158da59a.jpg)

Hi @lx-kika, 

Can you post a code snippet that calculates the different loss types you mention? 
Also, I think it should be fine to just use the first method to calculate the loss - I don't really see a reason as why to use the second method, no? 