Question about xla_spawn.py script and torch_xla.distributed.xla_multiprocessing
I am able to fine-tune a large BERT model using your `examples/xla_spawn.py` script, by calling it from a colab notebook shell. 

However, when I try essentially the same thing in a colab notebook, putting the code in a cell and calling 

    torch_xla.distributed.xla_multiprocessing.spawn((_mp_fn, start_method="fork")

also in a colab cell, I get errors that the TPUs have run out of memory, when trying to train. 

Is this because the start_method `fork` is less memory efficient? Or should this also work with "native" colab code?

I can give a MRE colab if it's helpful. 
I'm unaware of what might be causing this - maybe @sgugger has an answer for you!<|||||>I have confirmed that the issue is not with the `start_method="fork"`. 

Your `run_glue.py` script uses `HfArgumentParser` to set up the `training_args` parameter of the `Trainer`. If I instead set the `training_args` manually, I get the same errors on the TPUs as in the colab notebook, even though the script uses `start_method="spawn"`. 

I haven't attempted to figure out exactly what is being set in `training_args` to allow the large BERTSs to be trained on TPUs. <|||||>The training arguments handle the initialization logic for the distributed setup, so they should only be initiliazed inside the `_mp_fn` you launch in parallel.

To launch your training from a colab, you should check the `notebook_launcher` from Accelerate.<|||||>Yes, right. I found out about the training arguments after spending some time experimenting. 

I am using a python script, which is called from colab via the shell, and everything is working fine. 

I did try `accelerate` once before but could not get it working, but I create an new issue if I go back to that route and still have problems.