`logging_dir` is not being generated.
### System Info

Hi. I'm using huggingface model `bert base` on cloud TPU but it couldn't generate the `--logging_dir` that I expected.





### Who can help?

@sgugger 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

The script is at [here](https://github.com/GoogleCloudPlatform/ml-testing-accelerators/blob/master/tests/pytorch/nightly/hf-lm.libsonnet).

To reproduce, run these commands in a docker container on a TPU VM:
```
          $ cd
          $ git clone https://github.com/huggingface/transformers.git
          $ cd transformers && pip install .
          $ pip install datasets evaluate scikit-learn
          $ python3 examples/pytorch/xla_spawn.py \
            --num_cores 8 \
            examples/pytorch/language-modeling/run_mlm.py \
            --logging_dir ./tensorboard-metrics \
            --cache_dir ./cache_dir \
            --dataset_name wikitext \
            --dataset_config_name wikitext-2-raw-v1 \
            --do_train \
            --do_eval \
            --overwrite_output_dir \
            --output_dir language-modeling \
            --logging_steps 30 \
            --save_steps 3000 \
            --overwrite_cache \
            --debug tpu_metrics_debug \
           --model_type=bert \
          --model_name_or_path bert-base-cased \
          --num_train_epochs 1 \
          --per_device_train_batch_size 16 \
          --per_device_eval_batch_size 16
```


### Expected behavior

I expect a folder `tensorboard-metrics` to be generated in `~/transformers` but I couldn't find it.
Hi @vanbasten23, 

Do you have tensorboard installed in your environment?

Could you share the the running environment: run `transformers-cli env` in the terminal and copy-paste the output?<|||||>> Do you have tensorboard installed in your environment?

Thanks. It turns out once I installed tensorboard, the folder was generated. Btw, do you know how transformer writes to the tensorboard? I searched things like "from torch.utils.tensorboard import SummaryWriter" in the transformer codebase but I couldn't find any reference.
<|||||>Writing logic is defined in the [TensorBoardCallback](https://github.com/huggingface/transformers/blob/7edc33ac7a2572698045fed3b5115bca23f40805/src/transformers/integrations.py#L573). 

This is added by default as a reporting callback if tensorboard is in your environment [here](https://github.com/huggingface/transformers/blob/7edc33ac7a2572698045fed3b5115bca23f40805/src/transformers/trainer.py#L539). <|||||>Thanks a lot @amyeroberts !