GPT2 on TPU, training is so slow.
When training GPT2 on TPU from scratch, training loss is constant & evaluation loss is decreasing very small amount.



> [INFO|trainer.py:1776] 2021-03-26 04:06:22,551 >>   Num examples = 100000
> [INFO|trainer.py:1777] 2021-03-26 04:06:22,551 >>   Batch size = 2
> {'eval_loss': 4.687133312225342, 'eval_runtime': 736.3302, 'eval_samples_per_second': 135.809, 'epoch': 0.05}
>   1%|#                                                                                                     | 22000/2080235 [22:38:08<2499:54:52,  4.37s/it]                                                [INFO|trainer.py:1528] 2021-03-26 04:18:38,885 >> Saving model checkpoint to outputs/line_by_line/checkpoint-22000
> [INFO|configuration_utils.py:314] 2021-03-26 04:18:38,912 >> Configuration saved in outputs/line_by_line/checkpoint-22000/config.json
> [INFO|modeling_utils.py:837] 2021-03-26 04:18:56,125 >> Model weights saved in outputs/line_by_line/checkpoint-22000/pytorch_model.bin
> [INFO|tokenization_utils_base.py:1896] 2021-03-26 04:18:56,130 >> tokenizer config file saved in outputs/line_by_line/checkpoint-22000/tokenizer_config.json
> [INFO|tokenization_utils_base.py:1902] 2021-03-26 04:18:56,131 >> Special tokens file saved in outputs/line_by_line/checkpoint-22000/special_tokens_map.json
> {'loss': 2.56, 'learning_rate': 0.0004963706023598295, 'epoch': 0.05}
> {'loss': 2.56, 'learning_rate': 0.0004963465666138682, 'epoch': 0.05}
> {'loss': 2.56, 'learning_rate': 0.0004963225308679067, 'epoch': 0.05}
> {'loss': 2.56, 'learning_rate': 0.0004962984951219453, 'epoch': 0.05}
> {'loss': 2.56, 'learning_rate': 0.000496274459375984, 'epoch': 0.05}
> {'loss': 2.56, 'learning_rate': 0.0004962504236300226, 'epoch': 0.05}
> {'loss': 2.56, 'learning_rate': 0.0004962263878840611, 'epoch': 0.05}
> {'loss': 2.56, 'learning_rate': 0.0004962023521380998, 'epoch': 0.05}
> {'loss': 2.56, 'learning_rate': 0.0004961783163921384, 'epoch': 0.06}
> {'loss': 2.56, 'learning_rate': 0.000496154280646177, 'epoch': 0.06}
>   1%|#1                                                                                                    | 23000/2080235 [23:52:03<2524:57:42,  4.42s/it][INFO|trainer.py:1775] 2021-03-26 05:32:34,207 >> ***** Running Evaluation *****
> [INFO|trainer.py:1776] 2021-03-26 05:32:34,317 >>   Num examples = 100000
> [INFO|trainer.py:1777] 2021-03-26 05:32:34,317 >>   Batch size = 2
> {'eval_loss': 4.667241096496582, 'eval_runtime': 739.6907, 'eval_samples_per_second': 135.192, 'epoch': 0.06}`


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.