[scripts] Proposal: add a specific device flag
wdyt?

Will do in other scripts if this gets merged.

My use case is I have an instance with multiple GPUs and want to run one generation on `cuda:0`, another one on `cuda:1`, etc.
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1603?src=pr&el=h1) Report
> Merging [#1603](https://codecov.io/gh/huggingface/transformers/pull/1603?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/e16d46843a19ab289b82138e4eccec5610a76de7?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1603/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1603?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #1603   +/-   ##
=======================================
  Coverage   86.16%   86.16%           
=======================================
  Files          91       91           
  Lines       13593    13593           
=======================================
  Hits        11713    11713           
  Misses       1880     1880
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/1603?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [transformers/tokenization\_ctrl.py](https://codecov.io/gh/huggingface/transformers/pull/1603/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL3Rva2VuaXphdGlvbl9jdHJsLnB5) | `96.03% <0%> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1603?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1603?src=pr&el=footer). Last update [e16d468...b0af23c](https://codecov.io/gh/huggingface/transformers/pull/1603?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
<|||||>In my script, I take another approach for this. I assume that distributed training can only be instantiated by using torch's launch script 'or that at least the WORLD_SIZE env variable is set). `local_rank` will be used for the GPU id, even when not in distributed mode.

```python
# torch.distributed.launch adds a world_size environment variable
distributed = int(os.environ['WORLD_SIZE']) > 1 if 'WORLD_SIZE' in os.environ else False
```

Based on that, you can decide what you want to do with `local_rank`. If we're in distributed mode, start the process group, if we're not: use the `local_rank` cuda device.

```python
if local_rank == -1 or not torch.cuda.is_available():
    device = torch.device('cpu')
else:
    device = torch.device(f"cuda:{local_rank}")
    if distributed:
        dist.init_process_group(backend='nccl', init_method='env://')
```

As a bonus, to ensure that all processes such as validating only happen on the main device, even if that's not cuda:0 (even though personally I do that on all devices, too):

```python
is_first_process = not distributed or local_rank in [0, -1]
# ...
if args.do_eval and is_first_process:
    # do eval
```

I merely post this for possible inspiration, of course!<|||||>Sounds good to me let's add this to all the examples (and the template in `templates/adding_a_new_example_script`)<|||||>> Sounds good to me let's add this to all the examples

The other scripts maybe make less sense as you would want to train on all available devices? Not 100% sure yet.<|||||>Ok I see, then maybe let's have the device flag on `run_generation` instead of `run_squad` (as currently proposed in the PR)?<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
