TPU pod training with BERT 
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version:3.3.1
- Platform: TPU
- Python version: 
- PyTorch version (GPU?): 1.6 TPU v3-32
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?: Yes

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.
 
 albert, bert, GPT2, XLM: @LysandreJik 
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao 
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 blenderbot: @mariamabarham
 Bart: @sshleifer
 Marian: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao 
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->

## Information

Model I am using (Bert):

The problem arises when using:
* [x] the official example scripts: (give details below)
* [] my own modified scripts: (give details below)

https://huggingface.co/blog/how-to-train

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)


After starting training on TPU Pod using Pytorch XLA, I get following long message

```## ❓ Questions and Help
2020-10-14 14:14:33 10.164.0.60 [2] Traceback (most recent call last):
2020-10-14 14:14:33 10.164.0.60 [2]   File "/home/developer/XainBERT/train.py", line 111, in <module>
2020-10-14 14:14:33 10.164.0.60 [2]     prediction_loss_only=True,
2020-10-14 14:14:33 10.164.0.60 [2]   File "/anaconda3/envs/torch-xla-1.6/lib/python3.6/site-packages/transformers/trainer.py", line 248, in __init__
2020-10-14 14:14:33 10.164.0.60 [2]     self.model = model.to(args.device) if model is not None else None
2020-10-14 14:14:33 10.164.0.60 [2]   File "/anaconda3/envs/torch-xla-1.6/lib/python3.6/site-packages/transformers/file_utils.py", line 944, in wrapper
2020-10-14 14:14:33 10.164.0.60 [2]     return func(*args, **kwargs)
2020-10-14 14:14:33 10.164.0.60 [2]   File "/anaconda3/envs/torch-xla-1.6/lib/python3.6/site-packages/transformers/training_args.py", line 408, in device
2020-10-14 14:14:33 10.164.0.60 [2]     return self._setup_devices[0]
2020-10-14 14:14:33 10.164.0.60 [2]   File "/anaconda3/envs/torch-xla-1.6/lib/python3.6/site-packages/transformers/file_utils.py", line 934, in __get__
2020-10-14 14:14:33 10.164.0.60 [2]     cached = self.fget(obj)
2020-10-14 14:14:33 10.164.0.60 [2]   File "/anaconda3/envs/torch-xla-1.6/lib/python3.6/site-packages/transformers/file_utils.py", line 944, in wrapper
2020-10-14 14:14:33 10.164.0.60 [2]     return func(*args, **kwargs)
2020-10-14 14:14:33 10.164.0.60 [2]   File "/anaconda3/envs/torch-xla-1.6/lib/python3.6/site-packages/transformers/training_args.py", line 379, in _setup_devices
2020-10-14 14:14:33 10.164.0.60 [2]     device = xm.xla_device()
2020-10-14 14:14:33 10.164.0.60 [2]   File "/anaconda3/envs/torch-xla-1.6/lib/python3.6/site-packages/torch_xla/core/xla_model.py", line 167, in xla_device
2020-10-14 14:14:33 10.164.0.60 [2]     devkind=[devkind] if devkind is not None else None)
2020-10-14 14:14:33 10.164.0.60 [2]   File "/anaconda3/envs/torch-xla-1.6/lib/python3.6/site-packages/torch_xla/core/xla_model.py", line 72, in get_xla_supported_devices
2020-10-14 14:14:33 10.164.0.60 [2]     xla_devices = _DEVICES.value
2020-10-14 14:14:33 10.164.0.60 [2]   File "/anaconda3/envs/torch-xla-1.6/lib/python3.6/site-packages/torch_xla/utils/utils.py", line 30, in value
2020-10-14 14:14:33 10.164.0.60 [2]     self._value = self._gen_fn()
2020-10-14 14:14:33 10.164.0.60 [2]   File "/anaconda3/envs/torch-xla-1.6/lib/python3.6/site-packages/torch_xla/core/xla_model.py", line 18, in <lambda>
2020-10-14 14:14:33 10.164.0.60 [2]     _DEVICES = xu.LazyProperty(lambda: torch_xla._XLAC._xla_get_devices())
2020-10-14 14:14:33 10.164.0.60 [2] RuntimeError: tensorflow/compiler/xla/xla_client/xrt_computation_client.cc:244 : Check failed: default_device_target != options_.global_device_map.end() 
2020-10-14 14:14:33 10.164.0.60 [2] *** Begin stack trace ***
2020-10-14 14:14:33 10.164.0.60 [2]     tensorflow::CurrentStackTrace()
2020-10-14 14:14:33 10.164.0.60 [2]     xla::XrtComputationClient::XrtComputationClient(xla::XrtComputationClient::Options, std::unique_ptr<tensorflow::tpu::TopologyProto, std::default_delete<tensorflow::tpu::TopologyProto> >)
2020-10-14 14:14:33 10.164.0.60 [2]     xla::ComputationClient::Create()
2020-10-14 14:14:33 10.164.0.60 [2] 
2020-10-14 14:14:33 10.164.0.60 [2] 
2020-10-14 14:14:33 10.164.0.60 [2]     xla::ComputationClient::Get()
2020-10-14 14:14:33 10.164.0.60 [2] 
2020-10-14 14:14:33 10.164.0.60 [2] 
2020-10-14 14:14:33 10.164.0.60 [2]     _PyCFunction_FastCallDict
2020-10-14 14:14:33 10.164.0.60 [2] 
2020-10-14 14:14:33 10.164.0.60 [2]     _PyEval_EvalFrameDefault
2020-10-14 14:14:33 10.164.0.60 [2] 
2020-10-14 14:14:33 10.164.0.60 [2] 
2020-10-14 14:14:33 10.164.0.60 [2]     _PyEval_EvalFrameDefault
2020-10-14 14:14:33 10.164.0.60 [2]     PyEval_EvalCodeEx
2020-10-14 14:14:33 10.164.0.60 [2] 
2020-10-14 14:14:33 10.164.0.60 [2]     PyObject_Call
2020-10-14 14:14:33 10.164.0.60 [2] 
2020-10-14 14:14:33 10.164.0.60 [2]     _PyObject_GenericGetAttrWithDict
2020-10-14 14:14:33 10.164.0.60 [2]     _PyEval_EvalFrameDefault
2020-10-14 14:14:33 10.164.0.60 [2] 
2020-10-14 14:14:33 10.164.0.60 [2] 
2020-10-14 14:14:33 10.164.0.60 [2] 
2020-10-14 14:14:33 10.164.0.60 [2]     _PyEval_EvalFrameDefault
2020-10-14 14:14:33 10.164.0.60 [2] 
2020-10-14 14:14:33 10.164.0.60 [2] 
2020-10-14 14:14:33 10.164.0.60 [2]     _PyEval_EvalFrameDefault
2020-10-14 14:14:33 10.164.0.60 [2]     PyEval_EvalCodeEx
2020-10-14 14:14:33 10.164.0.60 [2] 
2020-10-14 14:14:33 10.164.0.60 [2]     PyObject_Call
2020-10-14 14:14:33 10.164.0.60 [2]     _PyEval_EvalFrameDefault
2020-10-14 14:14:33 10.164.0.60 [2] 
2020-10-14 14:14:33 10.164.0.60 [2] 
2020-10-14 14:14:33 10.164.0.60 [2] 
2020-10-14 14:14:33 10.164.0.60 [2]     _PyEval_EvalFrameDefault
2020-10-14 14:14:33 10.164.0.60 [2] 
2020-10-14 14:14:33 10.164.0.60 [2]     _PyFunction_FastCallDict
2020-10-14 14:14:33 10.164.0.60 [2]     _PyObject_FastCallDict
2020-10-14 14:14:33 10.164.0.60 [2]     PyObject_CallFunctionObjArgs
2020-10-14 14:14:33 10.164.0.60 [2]     _PyObject_GenericGetAttrWithDict
2020-10-14 14:14:33 10.164.0.60 [2]     _PyEval_EvalFrameDefault
2020-10-14 14:14:33 10.164.0.60 [2]     PyEval_EvalCodeEx
2020-10-14 14:14:33 10.164.0.60 [2] 
2020-10-14 14:14:33 10.164.0.60 [2]     PyObject_Call
2020-10-14 14:14:33 10.164.0.60 [2]     _PyEval_EvalFrameDefault
2020-10-14 14:14:33 10.164.0.60 [2]     PyEval_EvalCodeEx
2020-10-14 14:14:33 10.164.0.60 [2] 
2020-10-14 14:14:33 10.164.0.60 [2]     PyObject_Call
2020-10-14 14:14:33 10.164.0.60 [2] 
2020-10-14 14:14:33 10.164.0.60 [2]     _PyObject_GenericGetAttrWithDict
2020-10-14 14:14:33 10.164.0.60 [2]     _PyEval_EvalFrameDefault
2020-10-14 14:14:33 10.164.0.60 [2] 
2020-10-14 14:14:33 10.164.0.60 [2]     _PyFunction_FastCallDict
2020-10-14 14:14:33 10.164.0.60 [2]     _PyObject_FastCallDict
2020-10-14 14:14:33 10.164.0.60 [2]     _PyObject_Call_Prepend
2020-10-14 14:14:33 10.164.0.60 [2]     PyObject_Call
2020-10-14 14:14:33 10.164.0.60 [2] 
2020-10-14 14:14:33 10.164.0.60 [2] 
2020-10-14 14:14:33 10.164.0.60 [2]     _PyObject_FastCallDict
2020-10-14 14:14:33 10.164.0.60 [2]     _PyObject_FastCallKeywords
2020-10-14 14:14:33 10.164.0.60 [2] 
2020-10-14 14:14:33 10.164.0.60 [2]     _PyEval_EvalFrameDefault
2020-10-14 14:14:33 10.164.0.60 [2]     PyEval_EvalCodeEx
2020-10-14 14:14:33 10.164.0.60 [2]     PyEval_EvalCode
2020-10-14 14:14:33 10.164.0.60 [2] 
2020-10-14 14:14:33 10.164.0.60 [2]     PyRun_FileExFlags
2020-10-14 14:14:33 10.164.0.60 [2]     PyRun_SimpleFileExFlags
2020-10-14 14:14:33 10.164.0.60 [2]     Py_Main
2020-10-14 14:14:33 10.164.0.60 [2]     main
2020-10-14 14:14:33 10.164.0.60 [2]     __libc_start_main
2020-10-14 14:14:33 10.164.0.60 [2] 
2020-10-14 14:14:33 10.164.0.60 [2] *** End stack trace ***
020-10-14 14:27:11 10.164.0.61 [0] /anaconda3/envs/torch-xla-1.6/lib/python3.6/site-packages/transformers/trainer.py:267: FutureWarning: Passing `prediction_loss_only` as a keyword argument is deprecated and won't 
be possible in a future version. Use `args.prediction_loss_only` instead.
2020-10-14 14:27:11 10.164.0.61 [0]   FutureWarning,
2020-10-14 14:27:11 10.164.0.61 [0] Using deprecated `--per_gpu_train_batch_size` argument which will be removed in a future version. Using `--per_device_train_batch_size` is preferred.
2020-10-14 14:27:11 10.164.0.61 [0] Using deprecated `--per_gpu_train_batch_size` argument which will be removed in a future version. Using `--per_device_train_batch_size` is preferred.
2020-10-14 14:27:11 10.164.0.61 [0] Using deprecated `--per_gpu_train_batch_size` argument which will be removed in a future version. Using `--per_device_train_batch_size` is preferred.
Epoch:   0%|          | 0/1 [00:00<?, ?it/s]
2020-10-14 14:28:09 10.164.0.61 [0] 
2020-10-14 14:28:20 10.164.0.61 [0] 
2020-10-14 14:28:32 10.164.0.61 [0] 
2020-10-14 14:28:33 10.164.0.61 [0] 
2020-10-14 14:28:38 10.164.0.61 [0] 
2020-10-14 14:28:38 10.164.0.61 [0] 
2020-10-14 14:28:38 10.164.0.61 [0] 
2020-10-14 14:28:40 10.164.0.61 [0] 
2020-10-14 14:28:42 10.164.0.61 [0] 
2020-10-14 14:28:54 10.164.0.61 [0] 
2020-10-14 14:28:54 10.164.0.61 [0] 
2020-10-14 14:28:56 10.164.0.61 [0] 
2020-10-14 14:28:57 10.164.0.61 [0] 
2020-10-14 14:28:58 10.164.0.61 [0] 
2020-10-14 14:28:58 10.164.0.61 [0] 
2020-10-14 14:28:59 10.164.0.61 [0] 
2020-10-14 14:29:01 10.164.0.61 [0] 
2020-10-14 14:29:01 10.164.0.61 [0] 
2020-10-14 14:29:01 10.164.0.61 [0] 
2020-10-14 14:29:02 10.164.0.61 [0] 
2020-10-14 14:29:02 10.164.0.61 [0] 
```

I think the problem lies here 

```2020-10-14 14:14:33 10.164.0.60 [2] RuntimeError: tensorflow/compiler/xla/xla_client/xrt_computation_client.cc:244 : Check failed: default_device_target != options_.global_device_map.end() ```

But I do not understand what is wrong. 

I basically use the script from here https://huggingface.co/blog/how-to-train with some modifications, own data but Bert vocabulary. I have used the same script for training on TPUv3-8 as well which works fine.The transformer library version is 3.3.1 and I am using BERT tokeniser and BERT config. The distributed training on TPU pod worked with ImageNet example mentioned in https://cloud.google.com/tpu/docs/tutorials/pytorch-pod#configure-gcloud. 


## To reproduce
I can't really provide the whole script. But the training command is as follow:

python -m torch_xla.distributed.xla_dist  --tpu=$TPU_NAME --conda-env=torch-xla-1.6 -- python /home/developer/XainBERT/xla_spawn.py --num_cores 8 /home/developer/XainBERT/train.py 

Steps to reproduce the behavior:

1.
2.
3.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

Hi, I believe there is an error in your pytorch/xla install. I would follow the docs shared on [pytorch/xla](https://github.com/pytorch/xla) for best results.<|||||>Actually, I am using Google cloud for running the training job. And it comes already with torch-xla-1.6 environment and the imagenet example works file. 

As fas as I understand the problem is here https://github.com/pytorch/xla/blob/53eb3fc5e701cb8cc79a45dd0f79c6e585a27a41/torch_xla/core/xla_model.py#L18. 
I do not understand what is the problem though. <|||||>I don't really understand how you're launching your training. Our `xla_spawn.py` already takes care of parallelizing. I would do the following:

```
export TPU_IP_ADDRESS=xxx.xxx.xxx.xxx
export XRT_TPU_CONFIG="tpu_worker;0;$TPU_IP_ADDRESS:8470"

python xla_spawn.py --num_cores 8 /home/developer/XainBERT/train.py
```<|||||>This works when I am using 8 core TPU. However, when I try to use TPUv3-128, then I get the error about cluster resolver. So I follow the instructions from here https://cloud.google.com/tpu/docs/tutorials/pytorch-pod#configure-gcloud. Also, the PyTorch lightning website also refers to the same page https://pytorch-lightning.readthedocs.io/en/latest/tpu.html. <|||||>Ah, I see. Unfortunately, I don't have sufficient experience with TPU pods to help you debug this. Can you try to open an issue on pytorch/xla's github?<|||||>That's sad. Am I right, that Huggingface does not have TF implementation of Language models? I could try to switch to TF and try. Anyways, I have opened an issue on PyTorch/xla.<|||||>No, we do have TF implementations of language models. Most of our models are available in both PyTorch and TensorFlow.<|||||>However, this link says that there is not TFtrainer support for language-modelling using raw data.
https://huggingface.co/transformers/examples.html 

Is there any example available for TFtrainer?<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
