run_lm_finetuning
When I run_lm_finetuning with the exemplary training corpus (small_wiki_sentence_corpus.txt), I printed the tr_loss every 20 steps. I found that the tr_loss increases very fast. I wonder what the reason is.
![image](https://user-images.githubusercontent.com/40857896/53220384-35b0f400-369f-11e9-8902-27f97a1adacc.png)
![image](https://user-images.githubusercontent.com/40857896/53220402-4c574b00-369f-11e9-899b-c7962f28cde0.png)


What tr_loss are you exactly printing here? Is it possible that you just print this one here? https://github.com/huggingface/pytorch-pretrained-BERT/blob/2152bfeae82439600dc5b5deab057a3c4331c62d/examples/run_lm_finetuning.py#L600 If yes, you should divide it by the number of training steps (nb_tr_steps) first to get your average train loss. <|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
