Only access loss tensor every logging_steps
* tensor.item() was being called every step. This must not be done for XLA:TPU tensors as it's terrible for performance causing TPU<>CPU communication at each step. On RoBERTa MLM for example, it reduces step time by 30%, should be larger for smaller step time models/tasks.
* Train batch size was not correct in case a user uses the `per_gpu_train_batch_size` flag
* Avg reduce loss accross eval shards
* Log TPU debug metrics before last epoch break
Style issue will be solved with merge @sgugger <|||||>> Thanks for fixing this! I'd remove the first change in the logs though.

Thanks for the review! Done.<|||||># [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6802?src=pr&el=h1) Report
> Merging [#6802](https://codecov.io/gh/huggingface/transformers/pull/6802?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/9336086ab5d232cccd9512333518cf4299528882?el=desc) will **decrease** coverage by `0.42%`.
> The diff coverage is `89.47%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/6802/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/6802?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #6802      +/-   ##
==========================================
- Coverage   80.32%   79.89%   -0.43%     
==========================================
  Files         157      157              
  Lines       28589    28739     +150     
==========================================
- Hits        22963    22960       -3     
- Misses       5626     5779     +153     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/6802?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/\_\_init\_\_.py](https://codecov.io/gh/huggingface/transformers/pull/6802/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9fX2luaXRfXy5weQ==) | `99.28% <ø> (ø)` | |
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6802/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `82.66% <ø> (+0.25%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_flaubert.py](https://codecov.io/gh/huggingface/transformers/pull/6802/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9mbGF1YmVydC5weQ==) | `88.34% <ø> (+63.80%)` | :arrow_up: |
| [src/transformers/optimization.py](https://codecov.io/gh/huggingface/transformers/pull/6802/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9vcHRpbWl6YXRpb24ucHk=) | `82.28% <ø> (ø)` | |
| [src/transformers/tokenization\_t5.py](https://codecov.io/gh/huggingface/transformers/pull/6802/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fdDUucHk=) | `95.28% <ø> (-0.05%)` | :arrow_down: |
| [src/transformers/trainer.py](https://codecov.io/gh/huggingface/transformers/pull/6802/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90cmFpbmVyLnB5) | `53.23% <46.66%> (-0.43%)` | :arrow_down: |
| [...rc/transformers/data/datasets/language\_modeling.py](https://codecov.io/gh/huggingface/transformers/pull/6802/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9kYXRhL2RhdGFzZXRzL2xhbmd1YWdlX21vZGVsaW5nLnB5) | `90.69% <89.18%> (-1.14%)` | :arrow_down: |
| [src/transformers/data/data\_collator.py](https://codecov.io/gh/huggingface/transformers/pull/6802/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9kYXRhL2RhdGFfY29sbGF0b3IucHk=) | `91.90% <94.59%> (+2.19%)` | :arrow_up: |
| [src/transformers/configuration\_pegasus.py](https://codecov.io/gh/huggingface/transformers/pull/6802/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9jb25maWd1cmF0aW9uX3BlZ2FzdXMucHk=) | `100.00% <100.00%> (ø)` | |
| [src/transformers/data/datasets/\_\_init\_\_.py](https://codecov.io/gh/huggingface/transformers/pull/6802/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9kYXRhL2RhdGFzZXRzL19faW5pdF9fLnB5) | `100.00% <100.00%> (ø)` | |
| ... and [25 more](https://codecov.io/gh/huggingface/transformers/pull/6802/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/6802?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6802?src=pr&el=footer). Last update [9336086...2b981cd](https://codecov.io/gh/huggingface/transformers/pull/6802?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
