[s2s] distributed eval gets stuck on error w/ multigpu
`examples/seq2seq/distillation.py` and probably others remain hanging on internal error when run w/ multiple gpus (2 here):

```
rm -r /tmp/tmpqajqhzwo; PYTHONPATH="src" python examples/seq2seq/distillation.py --supervise_forward --normalize_hidden --label_smoothing=0.0 --eval_beams=1 --val_metric=loss --save_top_k=1 --adafactor --early_stopping_patience=-1 --logger_name=default --length_penalty=0.5 --cache_dir= --task=summarization --num_workers=2 --alpha_hid=0 --freeze_embeds --sortish_sampler --student_decoder_layers=1 --val_check_interval=0.5 --output_dir=/tmp/tmpqajqhzwo --no_teacher --fp16_opt_level=O1 --gpus=2 --max_grad_norm=1.0 --do_train --do_predict --accumulate_grad_batches=1 --seed=42 --model_name_or_path=sshleifer/tinier_bart --config_name= --tokenizer_name=facebook/bart-large --learning_rate=0.3 --lr_scheduler=linear --weight_decay=0.0 --adam_epsilon=1e-08 --warmup_steps=0 --max_epochs=2 --train_batch_size=1 --eval_batch_size=2 --max_source_length=12 --max_target_length=12 --val_max_target_length=12 --test_max_target_length=12 --n_train=-1 --n_val=-1 --n_test=-1 --student_encoder_layers=1 --freeze_encoder --data_dir=examples/seq2seq/test_data/wmt_en_ro --alpha_mlm=0.2 --alpha_ce=0.8 --teacher=sshleifer/bart-tiny-random
```

last output:
```
initializing ddp: GLOBAL_RANK: 0, MEMBER: 1/2
Traceback (most recent call last):
  File "/mnt/nvme1/code/huggingface/transformers-master/examples/seq2seq/distillation.py", line 281, in <module>
    distill_main(args)
  File "/mnt/nvme1/code/huggingface/transformers-master/examples/seq2seq/distillation.py", line 269, in distill_main
    check_output_dir(args, expected_items=3)
  File "/mnt/nvme1/code/huggingface/transformers-master/examples/seq2seq/utils.py", line 641, in check_output_dir
    raise ValueError(
ValueError: Output directory (/tmp/tmpqajqhzwo) already exists and has 7 items in it (expected 3 items). Use --overwrite_output_dir to overcome.
```
and now it hangs, holding onto the gpu. Can't even Ctrl-C the process - needed to suspend+kill manually.

I know that adding `--overwrite_output_dir` will remove the error, but this is not the issue. It shouldn't hang on error (e.g. the test suite needs to continue running in such event).

@sshleifer 
same happens with `finetune.py` - happened in another run when it hit OOM. So basically any error.<|||||>@williamFalcon @SeanNaren (lightning friends)

Do you guys have a clever way to collect failures in your multigpu tests?
When something breaks, our multigpu test hangs.
<|||||>yes... good questions haha. 

So, some things we know:

1. multi gpu tests should run one per test (ie: don’t parametrize via pytest). Seems that the way pytest starts an experiment does not play well with pytorch distributed. 

2. ddp in lightning needs to use subprocess inside a test and call an external file. 
https://github.com/PyTorchLightning/pytorch-lightning/blob/master/tests/backends/test_ddp.py#L57

3. ddp spawn tests need to adhere to that single test per function call i mentioned in 1. pytest parametrized ddp testss WILL freeze the build. <|||||>Thank you for the insights, @williamFalcon 

That is the case already - I discovered the subprocess idea by looking at your distributed ddp test ;) And none of these are parametrized.

So it must be something else.

p.s. btw, have you tried the `parameterized` module https://pypi.org/project/parameterized/? It's more flexible than `pytest`'s `parametrize` - perhaps it won't have the same impact (but that's unrelated to this issue).<|||||>oh, and to clarify, this has nothing to do with testing. The hanging happens in the standalone scripts. <|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
