training_loss in TFTrainer
# What does this PR do?
The purpose of ```training_loss``` in``` TFTrainer``` is logging.
However, ```training_loss``` shows a very large number while it decreases.
And it is doubled with twice the ```gradient_accumulation_steps```

1. ```training_loss``` is accumulated during epochs. Now, it is only calculated in a step.
2. Like ```Trainer```, ```training_loss``` in ```TFTrainer``` considers ```n_replicas``` and ```gradient_accumulation_steps```.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
 tensorflow: @jplu
Here the logs of a training with
```
python run_tf_glue.py --task_name mrpc --model_name_or_path bert-base-cased --output_dir model --num_train_epochs 4 --per_device_train_batch_size 16 --per_device_eval_batch_size 16 --do_train --do_eval --do_predict --logging_steps 10 --overwrite_output_dir --gradient_accumulation_steps 2
```

```
[INFO|trainer_tf.py:522] 2021-01-15 10:27:59,116 >> ***** Running training *****
[INFO|trainer_tf.py:523] 2021-01-15 10:27:59,123 >>   Num examples = 3668
[INFO|trainer_tf.py:525] 2021-01-15 10:27:59,124 >>   Num Epochs = 4
[INFO|trainer_tf.py:526] 2021-01-15 10:27:59,124 >>   Instantaneous batch size per device = 16
[INFO|trainer_tf.py:527] 2021-01-15 10:27:59,124 >>   Total train batch size (w. parallel, distributed & accumulation) = 32
[INFO|trainer_tf.py:530] 2021-01-15 10:27:59,125 >>   Gradient Accumulation steps = 2
[INFO|trainer_tf.py:531] 2021-01-15 10:27:59,136 >>   Steps per epoch = 115
[INFO|trainer_tf.py:532] 2021-01-15 10:27:59,137 >>   Total optimization steps = 460
[INFO|trainer_tf.py:398] 2021-01-15 10:28:50,371 >> {'loss': 0.6347228, 'learning_rate': 4.891304e-05, 'epoch': 0.08695652173913043, 'step': 10}
[INFO|trainer_tf.py:398] 2021-01-15 10:28:56,791 >> {'loss': 0.604829, 'learning_rate': 4.7826084e-05, 'epoch': 0.17391304347826086, 'step': 20}
[INFO|trainer_tf.py:398] 2021-01-15 10:29:03,192 >> {'loss': 0.62615454, 'learning_rate': 4.673913e-05, 'epoch': 0.2608695652173913, 'step': 30}
[INFO|trainer_tf.py:398] 2021-01-15 10:29:09,614 >> {'loss': 0.61436784, 'learning_rate': 4.5652174e-05, 'epoch': 0.34782608695652173, 'step': 40}
[INFO|trainer_tf.py:398] 2021-01-15 10:29:16,163 >> {'loss': 0.60542804, 'learning_rate': 4.456522e-05, 'epoch': 0.43478260869565216, 'step': 50}
[INFO|trainer_tf.py:398] 2021-01-15 10:29:22,633 >> {'loss': 0.60221016, 'learning_rate': 4.347826e-05, 'epoch': 0.5217391304347826, 'step': 60}
[INFO|trainer_tf.py:398] 2021-01-15 10:29:29,129 >> {'loss': 0.59315145, 'learning_rate': 4.2391304e-05, 'epoch': 0.6086956521739131, 'step': 70}
[INFO|trainer_tf.py:398] 2021-01-15 10:29:35,655 >> {'loss': 0.5896678, 'learning_rate': 4.1304345e-05, 'epoch': 0.6956521739130435, 'step': 80}
[INFO|trainer_tf.py:398] 2021-01-15 10:29:42,209 >> {'loss': 0.5796127, 'learning_rate': 4.0217386e-05, 'epoch': 0.782608695652174, 'step': 90}
[INFO|trainer_tf.py:398] 2021-01-15 10:29:48,779 >> {'loss': 0.5678522, 'learning_rate': 3.9130435e-05, 'epoch': 0.8695652173913043, 'step': 100}
[INFO|trainer_tf.py:398] 2021-01-15 10:29:55,365 >> {'loss': 0.55807614, 'learning_rate': 3.8043476e-05, 'epoch': 0.9565217391304348, 'step': 110}
[INFO|trainer_tf.py:398] 2021-01-15 10:30:04,348 >> {'loss': 0.32373077, 'learning_rate': 3.695652e-05, 'epoch': 1.0434782608695652, 'step': 120}
[INFO|trainer_tf.py:398] 2021-01-15 10:30:10,920 >> {'loss': 0.3261666, 'learning_rate': 3.5869565e-05, 'epoch': 1.1304347826086956, 'step': 130}
[INFO|trainer_tf.py:398] 2021-01-15 10:30:17,516 >> {'loss': 0.34052417, 'learning_rate': 3.478261e-05, 'epoch': 1.2173913043478262, 'step': 140}
[INFO|trainer_tf.py:398] 2021-01-15 10:30:24,125 >> {'loss': 0.35018474, 'learning_rate': 3.369565e-05, 'epoch': 1.3043478260869565, 'step': 150}
[INFO|trainer_tf.py:398] 2021-01-15 10:30:30,714 >> {'loss': 0.35887596, 'learning_rate': 3.260869e-05, 'epoch': 1.391304347826087, 'step': 160}
[INFO|trainer_tf.py:398] 2021-01-15 10:30:37,303 >> {'loss': 0.34891757, 'learning_rate': 3.1521737e-05, 'epoch': 1.4782608695652173, 'step': 170}
[INFO|trainer_tf.py:398] 2021-01-15 10:30:43,900 >> {'loss': 0.33256933, 'learning_rate': 3.0434781e-05, 'epoch': 1.5652173913043477, 'step': 180}
[INFO|trainer_tf.py:398] 2021-01-15 10:30:50,481 >> {'loss': 0.32668048, 'learning_rate': 2.934782e-05, 'epoch': 1.6521739130434783, 'step': 190}
[INFO|trainer_tf.py:398] 2021-01-15 10:30:57,079 >> {'loss': 0.31888676, 'learning_rate': 2.8260865e-05, 'epoch': 1.7391304347826086, 'step': 200}
[INFO|trainer_tf.py:398] 2021-01-15 10:31:03,688 >> {'loss': 0.31276095, 'learning_rate': 2.7173912e-05, 'epoch': 1.8260869565217392, 'step': 210}
[INFO|trainer_tf.py:398] 2021-01-15 10:31:10,284 >> {'loss': 0.30366346, 'learning_rate': 2.6086956e-05, 'epoch': 1.9130434782608696, 'step': 220}
[INFO|trainer_tf.py:398] 2021-01-15 10:31:16,885 >> {'loss': 0.2903903, 'learning_rate': 2.5e-05, 'epoch': 2.0, 'step': 230}
[INFO|trainer_tf.py:398] 2021-01-15 10:31:26,095 >> {'loss': 0.15675393, 'learning_rate': 2.3913042e-05, 'epoch': 2.0869565217391304, 'step': 240}
[INFO|trainer_tf.py:398] 2021-01-15 10:31:32,671 >> {'loss': 0.14483282, 'learning_rate': 2.2826087e-05, 'epoch': 2.1739130434782608, 'step': 250}
[INFO|trainer_tf.py:398] 2021-01-15 10:31:39,275 >> {'loss': 0.14147088, 'learning_rate': 2.173913e-05, 'epoch': 2.260869565217391, 'step': 260}
[INFO|trainer_tf.py:398] 2021-01-15 10:31:45,866 >> {'loss': 0.13758971, 'learning_rate': 2.0652174e-05, 'epoch': 2.3478260869565215, 'step': 270}
[INFO|trainer_tf.py:398] 2021-01-15 10:31:52,464 >> {'loss': 0.13357341, 'learning_rate': 1.9565217e-05, 'epoch': 2.4347826086956523, 'step': 280}
[INFO|trainer_tf.py:398] 2021-01-15 10:31:59,049 >> {'loss': 0.12877393, 'learning_rate': 1.8478258e-05, 'epoch': 2.5217391304347827, 'step': 290}
[INFO|trainer_tf.py:398] 2021-01-15 10:32:05,682 >> {'loss': 0.13753517, 'learning_rate': 1.7391301e-05, 'epoch': 2.608695652173913, 'step': 300}
[INFO|trainer_tf.py:398] 2021-01-15 10:32:12,281 >> {'loss': 0.1319594, 'learning_rate': 1.6304344e-05, 'epoch': 2.6956521739130435, 'step': 310}
[INFO|trainer_tf.py:398] 2021-01-15 10:32:18,883 >> {'loss': 0.12644322, 'learning_rate': 1.5217389e-05, 'epoch': 2.782608695652174, 'step': 320}
[INFO|trainer_tf.py:398] 2021-01-15 10:32:25,472 >> {'loss': 0.12481367, 'learning_rate': 1.41304345e-05, 'epoch': 2.869565217391304, 'step': 330}
[INFO|trainer_tf.py:398] 2021-01-15 10:32:32,082 >> {'loss': 0.12073966, 'learning_rate': 1.3043478e-05, 'epoch': 2.9565217391304346, 'step': 340}
[INFO|trainer_tf.py:398] 2021-01-15 10:32:41,403 >> {'loss': 0.10288413, 'learning_rate': 1.1956521e-05, 'epoch': 3.0434782608695654, 'step': 350}
[INFO|trainer_tf.py:398] 2021-01-15 10:32:47,955 >> {'loss': 0.09858045, 'learning_rate': 1.0869565e-05, 'epoch': 3.130434782608696, 'step': 360}
[INFO|trainer_tf.py:398] 2021-01-15 10:32:54,532 >> {'loss': 0.07963112, 'learning_rate': 9.782609e-06, 'epoch': 3.217391304347826, 'step': 370}
[INFO|trainer_tf.py:398] 2021-01-15 10:33:01,104 >> {'loss': 0.08428383, 'learning_rate': 8.6956525e-06, 'epoch': 3.3043478260869565, 'step': 380}
[INFO|trainer_tf.py:398] 2021-01-15 10:33:07,684 >> {'loss': 0.0844244, 'learning_rate': 7.6086967e-06, 'epoch': 3.391304347826087, 'step': 390}
[INFO|trainer_tf.py:398] 2021-01-15 10:33:14,284 >> {'loss': 0.08690852, 'learning_rate': 6.5217405e-06, 'epoch': 3.4782608695652173, 'step': 400}
[INFO|trainer_tf.py:398] 2021-01-15 10:33:20,877 >> {'loss': 0.0832295, 'learning_rate': 5.434781e-06, 'epoch': 3.5652173913043477, 'step': 410}
[INFO|trainer_tf.py:398] 2021-01-15 10:33:27,494 >> {'loss': 0.078029804, 'learning_rate': 4.3478244e-06, 'epoch': 3.6521739130434785, 'step': 420}
[INFO|trainer_tf.py:398] 2021-01-15 10:33:34,095 >> {'loss': 0.079320244, 'learning_rate': 3.2608687e-06, 'epoch': 3.7391304347826084, 'step': 430}
[INFO|trainer_tf.py:398] 2021-01-15 10:33:40,709 >> {'loss': 0.076877564, 'learning_rate': 2.1739122e-06, 'epoch': 3.8260869565217392, 'step': 440}
[INFO|trainer_tf.py:398] 2021-01-15 10:33:47,324 >> {'loss': 0.07551385, 'learning_rate': 1.0869561e-06, 'epoch': 3.9130434782608696, 'step': 450}
[INFO|trainer_tf.py:398] 2021-01-15 10:33:53,941 >> {'loss': 0.07157838, 'learning_rate': 0.0, 'epoch': 4.0, 'step': 460}
```

Nothing seems wrong in the loss computation.

```
eval_acc = 0.8518518518518519
eval_f1 = 0.8954248366013072
eval_acc_and_f1 = 0.8736383442265796
```<|||||>@jplu 
Yes, you are right, and I am wrong. 
My dataset format was wrong (```labels``` in dataset for ```TFGPT2LMHead``` should be ```tensor```, but was ```dict```  yesterday). Sorry for the confusion.

However, there is one problem though. ```training_loss``` is not properly calculated with successive training. Run ```run_tf_glue.py``` with ```save_steps```. I train 40 steps, and train again from that ckpt. Results are shown below, where loss increases and decreases.

```
[INFO|trainer_tf.py:398] 2021-01-15 16:19:09,011 >> {'loss': 0.10616198, 'learning_rate': 4.456522e-05, 'epoch': 0.43478260869565216, 'step': 50}
[INFO|trainer_tf.py:398] 2021-01-15 16:19:20,271 >> {'loss': 0.17419913, 'learning_rate': 4.347826e-05, 'epoch': 0.5217391304347826, 'step': 60}
[INFO|trainer_tf.py:398] 2021-01-15 16:19:33,935 >> {'loss': 0.2174806, 'learning_rate': 4.2391304e-05, 'epoch': 0.6086956521739131, 'step': 70}
[INFO|trainer_tf.py:398] 2021-01-15 16:19:47,342 >> {'loss': 0.25698015, 'learning_rate': 4.1304345e-05, 'epoch': 0.6956521739130435, 'step': 80}
[INFO|trainer_tf.py:398] 2021-01-15 16:20:04,633 >> {'loss': 0.28348362, 'learning_rate': 4.0217386e-05, 'epoch': 0.782608695652174, 'step': 90}
[INFO|trainer_tf.py:398] 2021-01-15 16:20:22,000 >> {'loss': 0.29971126, 'learning_rate': 3.9130435e-05, 'epoch': 0.8695652173913043, 'step': 100}
[INFO|trainer_tf.py:398] 2021-01-15 16:20:35,515 >> {'loss': 0.30674613, 'learning_rate': 3.8043476e-05, 'epoch': 0.9565217391304348, 'step': 110}
[INFO|trainer_tf.py:398] 2021-01-15 16:20:57,368 >> {'loss': 0.3384542, 'learning_rate': 3.695652e-05, 'epoch': 1.0434782608695652, 'step': 120}
[INFO|trainer_tf.py:398] 2021-01-15 16:21:10,066 >> {'loss': 0.25661522, 'learning_rate': 3.5869565e-05, 'epoch': 1.1304347826086956, 'step': 130}
[INFO|trainer_tf.py:398] 2021-01-15 16:21:23,626 >> {'loss': 0.2714903, 'learning_rate': 3.478261e-05, 'epoch': 1.2173913043478262, 'step': 140}
[INFO|trainer_tf.py:398] 2021-01-15 16:21:42,122 >> {'loss': 0.262019, 'learning_rate': 3.369565e-05, 'epoch': 1.3043478260869565, 'step': 150}
[INFO|trainer_tf.py:398] 2021-01-15 16:21:55,652 >> {'loss': 0.27134755, 'learning_rate': 3.260869e-05, 'epoch': 1.391304347826087, 'step': 160}
```

This comes from  ```steps_trained_in_current_epoch``` and ```training_loss = self.train_loss.result() / (step + 1)``` in ```TFTrainer```.
For 41 step (1 step after running from cpkt-40 in this case), only one loss is accumulated but step is 40.

I simply revise it by initially saving ```steps_trained_in_current_epoch``` to another constant. 
This may be treated in different PR.<|||||>Humm looks to be an issue indeed. You can keep this PR open to fix this, feel free to ask questions if I can help :)