Adding Minimal Reproducible Usage Example For TPU support on examples/seq2seq
Attempt to resolve https://github.com/huggingface/transformers/issues/5895.

Minimal Working [Colab Example](https://colab.research.google.com/drive/16q2GWrnZ0Tjg1OxJQUcaWKCWwn3Jh5z0?usp=sharing).

For using more than a single core, one needs to ensure that enough RAM is available else wait for PyTorch-XLA to release a stable version. They have also [released](https://github.com/pytorch/xla/issues/1870#issuecomment-623603323) a fix way back that prevent excessive memory usage for nightly.
The test will obviously break, right?<|||||>Any progress on merging this?<|||||>they won't accept it as the checks have failed? 
But it's expected for the checks to fail as I have modified `modeling_bart` and it's gonna have one more Param count.<|||||>Is that so @sshleifer?<|||||>Thanks for the contribution, this looks awesome!

We can't merge with failing tests, but I think the tests can pass.

Could you also check
```
RUN_SLOW=1 pytest tests/test_modeling_bart.py
RUN_SLOW=1 pytest tests/test_modeling_marian.py
RUN_SLOW=1 pytest test_modeling_mbart.py
```
add the `USE_CUDA=1` prefix to make them run faster on GPU.<|||||>Actually can we add a `support_tpu` flag to BartConfig, init it to False, and only allocate `lm_head` if it's set to True. I'm concerned that we are wasting RAM when we train on GPU. (I would happily change my mind if I encountered evidence that this change doesn't change GPU RAM consumption.)<|||||>I tried this version and it seems to work but it stucks at "Validation sanity check". Working colab [here](https://colab.research.google.com/drive/1NA9_EPEBNmo7feQ60iiznsLP_XbWwQmC?usp=sharing)<|||||>Well I removed the validation check altogether by passing in the concerned
flag to 0. Tried debugging to find out what's causing it, but I couldn't
figure it out.

If you will train and validate, it will work.

On Mon, 27 Jul 2020, 17:46 marton-avrios, <notifications@github.com> wrote:

> I tried this version and it seems to work but it stucks at "Validation
> sanity check".
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/pull/5960#issuecomment-664360705>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AFNPJJQYIPQDG5ZPYXDTTT3R5VV23ANCNFSM4PEHWJYA>
> .
>
<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>This is now supported by `Seq2SeqTrainer`. Use that if you want TPU support!