[s2s] test_finetune_trainer_slow fails when run in group
On dual gpu when running `test_finetune_trainer.py::TestFinetuneTrainer::test_finetune_trainer_slow` alone - all is good.

when running it with all the other tests in that file it fails:

```
RUN_SLOW=1 pytest -sv test_finetune_trainer.py
[...]
self = <seq2seq.test_finetune_trainer.TestFinetuneTrainer testMethod=test_finetune_trainer_slow>

    @slow
    def test_finetune_trainer_slow(self):
        # There is a missing call to __init__process_group somewhere
        output_dir = self.run_trainer(
            eval_steps=2, max_len="128", model_name=MARIAN_MODEL, num_train_epochs=10, distributed=False
        )

        # Check metrics
        logs = TrainerState.load_from_json(os.path.join(output_dir, "trainer_state.json")).log_history
        eval_metrics = [log for log in logs if "eval_loss" in log.keys()]
        first_step_stats = eval_metrics[0]
        last_step_stats = eval_metrics[-1]

>       assert first_step_stats["eval_bleu"] < last_step_stats["eval_bleu"]  # model learned nothing
E       AssertionError: assert 0.0 < 0.0

test_finetune_trainer.py:130: AssertionError
----------------------------------------------------------------------- Captured log call -----------------------------------------------------------------------
WARNING  seq2seq.finetune_trainer:finetune_trainer.py:160 Process rank: -1, device: cuda:0, n_gpu: 2, distributed training: False, 16-bits training: False

==================================================================== short test summary info ====================================================================
FAILED test_finetune_trainer.py::TestFinetuneTrainer::test_finetune_trainer_slow - AssertionError: assert 0.0 < 0.0
================================================ 1 failed, 7 passed, 1 skipped, 17 warnings in 102.82s (0:01:42) ================================================
```

For some reason it fails to learn anything when some other tests run before it. 

tested with pytorch-nightly + py38.

I also have the same failure on one GPU on my side FYI (but no failure when run on its own).<|||||>Thank you, @sgugger  - your input helped a lot to reduce the sequence quickly! So this sequence fails:
```
CUDA_VISIBLE_DEVICES=0 RUN_SLOW=1 pytest \
test_finetune_trainer.py::TestFinetuneTrainer::test_finetune_trainer_apex \ 
test_finetune_trainer.py::TestFinetuneTrainer::test_finetune_trainer_slow
```
Something about apex.

This sequence with another similar test before it but no apex doesn't fail:
```
CUDA_VISIBLE_DEVICES=0 RUN_SLOW=1 pytest \
test_finetune_trainer.py::TestFinetuneTrainer::test_finetune_trainer_no_dist \
test_finetune_trainer.py::TestFinetuneTrainer::test_finetune_trainer_slow
```<|||||>This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.