training text_classification with tpu using xla_spawn gives wrong result
I tested text_classification code with tpu and gpu versions shown below.
tpu(colab with 8cores) and gpu(colab) versions take 2min and 17min, respectively, which is nice.
Training gpu version gives good behavior, where loss decreases continuously.

However, for tpu version, the dataset is almost divided into 8 segments(so 1/8 times consuming), but each node may not be connected. Results from tpu of each node are just result with gpu with 1/8 of the original dataset size.

1.
Should I change something?
Or if I`d like to use TPU, I have to use TF version?

2.
My final goal is to train Roberta on TPU in Korean.
There are three options.
1. Huggingface Trainer with xla  --> I am here.
2. Huggingface TFTrainer --> TFTrainer supports TPU, but need to make mlm datasets.
3. Fairseq with xla
If there are any sources or examples, please let me know.

Thanks,

# tpu version (same one shown in the document 'https://github.com/huggingface/transformers/tree/master/examples/text-classification'):
python examples/xla_spawn.py \
  --num_cores=8 \
  transformers/examples/text-classification/run_glue.py \
  --do_train \
  --do_eval \
  --task_name=mrpc \
  --num_train_epochs=3 \
  --max_seq_length=128 \
  --learning_rate=5e-5 \
  --output_dir=/tmp/mrpc \
  --overwrite_output_dir \
  --logging_steps=5 \
  --save_steps=5 \
  --tpu_metrics_debug \
  --model_name_or_path=bert-base-cased \
  --per_device_train_batch_size=64 \
  --per_device_eval_batch_size=64

# single gpu version (remove   --num_cores=8,   --tpu_metrics_debug)
python examples/xla_spawn.py \
  transformers/examples/text-classification/run_glue.py \
  --do_train \
  --do_eval \
  --task_name=mrpc \
  --num_train_epochs=3 \
  --max_seq_length=128 \
  --learning_rate=5e-5 \
  --output_dir=/tmp/mrpc \
  --overwrite_output_dir \
  --logging_steps=5 \
  --save_steps=5 \
  --model_name_or_path=bert-base-cased \
  --per_device_train_batch_size=64 \
  --per_device_eval_batch_size=64
Hello! I'm not sure I completely understand the issue. Are you saying you do not obtain the results you expected when using the 8 cores of the TPU, vs using a single TPU core?<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.