[testing] concurrent dist tests fail when using the same master_port
The failing `tests/extended/test_trainer_ext.py::TestTrainerExt::test_run_seq2seq_ddp` on multi-gpu slow runner happens because of concurrency of `-n 2` on push (it's `-n 1` on scheduled), so we end up with 2 dist tests running at the same time and clashing on ending up using the same default port used by `torch.distributed.launch`.  and the latter test failing with "Address already in Use".

I started a discussion at https://github.com/pytorch/pytorch/issues/59978 hoping to expose the `init_method=file://` through the CLI, but alas it won't help since the test suite needs to support the older pytorch even if it's exposed.

Furthermore, pytorch-1.9.1 or perhaps higher will have a different way using FileStore as a "rendezvous endpoint" in `torch.distributed.run` - a replacement for `torch.distributed.launch`.

Meanwhile it was proposed to use TorchElastic, which requires launching a server https://pytorch.org/elastic/0.2.2/quickstart.html before the test suite start and somehow ensuring it gets killed at the end. This looks very error-prone to me, especially when the test suite fails.

But I'm not yet sure how to come up with an algorithm to get the unique unused port to each test client, other than writing yet another server that will do the port management on demand.

@LysandreJik 

Would it be helpful to extract the tests that use the same port with a custom decorator, and run them in a separate `run` directive with `-n 1`?<|||||>That is a possibility too, see the simple proposed solution https://github.com/huggingface/transformers/pull/12166 - perhaps to try first.