<h1 dir="auto"><g-emoji class="g-emoji" alias="rocket" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f680.png">🚀</g-emoji> Feature request</h1>
<p dir="auto">Modify BERT models (src/transformers/modeling_bert.py) to conform to TorchScript requirements, so they can be <code class="notranslate">jit.script()</code>-ed, not just <code class="notranslate">jit.trace()</code>-ed (as is <a href="https://huggingface.co/transformers/torchscript.html" rel="nofollow">currently the only supported option</a>)</p>
<p dir="auto"><em>Note:</em> I have a working version implementing this, which I would like to contribute.<br>
See below.</p>
<h2 dir="auto">Motivation</h2>
<p dir="auto">A scriptable model would allow for variable-length input, offering big speedup gains and simplification (no need to create different models for different input lengths).</p>
<p dir="auto">In addition, it would avoid other potential pitfalls with tracing (e.g., code paths that are input dependent and not covered by the tracing example input).</p>
<p dir="auto">Related issues:<br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="545733209" data-permission-text="Title is private" data-url="https://github.com/huggingface/transformers/issues/2417" data-hovercard-type="issue" data-hovercard-url="/huggingface/transformers/issues/2417/hovercard" href="https://github.com/huggingface/transformers/issues/2417">#2417</a><br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="489632828" data-permission-text="Title is private" data-url="https://github.com/huggingface/transformers/issues/1204" data-hovercard-type="issue" data-hovercard-url="/huggingface/transformers/issues/1204/hovercard" href="https://github.com/huggingface/transformers/issues/1204">#1204</a><br>
possibly also<br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="505032772" data-permission-text="Title is private" data-url="https://github.com/huggingface/transformers/issues/1477" data-hovercard-type="issue" data-hovercard-url="/huggingface/transformers/issues/1477/hovercard" href="https://github.com/huggingface/transformers/issues/1477">#1477</a><br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="472804028" data-permission-text="Title is private" data-url="https://github.com/huggingface/transformers/issues/902" data-hovercard-type="issue" data-hovercard-url="/huggingface/transformers/issues/902/hovercard" href="https://github.com/huggingface/transformers/issues/902">#902</a></p>
<h2 dir="auto">Your contribution</h2>
<p dir="auto">I have a working PR that modifies all the models in src/transformers/modeling_bert.py and makes them TorchScript-able. I have not tested it on other models that use BERT components (e.g., albert), but it should be possible to expand the capability to those, as well.</p>
<p dir="auto">However, it would require some significant work to make it ready for submission: besides formatting, documentation, testing etc., my current version changes the method signatures, and I would need to avoid that to maintain backward-compatibility.</p>
<p dir="auto">Before putting in that work, I'd like to make sure that such a PR is something you'd be interested in and would be willing to merge in, assuming it meets the requirements.</p>
<p dir="auto"><code class="notranslate">tests/test_trainer_distributed.py</code> fails occasionally on multi-gpu github runner CI and as a result doesn't free up the 29500 default distributed port.</p>
<p dir="auto">This could be caused by an occasional deadlock discusses in testing_utils.py's <code class="notranslate">_stream_subprocess``. When debugging one such zombie it was stuck in </code>exec(eval(sys.stdin.readline()))`</p>
<p dir="auto">Note that other similar tests under <code class="notranslate">examples</code> don't exhibit the same behavior - perhaps it somehow has to do with this being a different script that it runs (this test runs its own file as the distributed script).</p>
<p dir="auto">The bt of the subsequent failures is long and confusing, as there are several mixed failures, but it's all really one failure: <code class="notranslate">Address already in use</code> since the previous distributed run of the same test didn't free up this port.</p>
<p dir="auto">A quick check should show which process is bound to it:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="netstat -tulpn | grep :29500"><pre class="notranslate"><code class="notranslate">netstat -tulpn | grep :29500
</code></pre></div>
<p dir="auto">The full bt:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="NCCL_DEBUG=INFO pytest -sv tests/test_trainer_distributed.py
================================================================ test session starts ================================================================
platform linux -- Python 3.7.4, pytest-6.2.2, py-1.10.0, pluggy-0.13.1 -- /home/github_actions/actions-runner/_work/transformers/transformers/.env/bin/python
cachedir: .pytest_cache
rootdir: /home/github_actions/actions-runner/_work/transformers/transformers
plugins: xdist-2.2.1, forked-1.3.0
collected 1 item                                                                                                                                    

tests/test_trainer_distributed.py::TestTrainerDistributed::test_trainer 
Running:  /home/github_actions/actions-runner/_work/transformers/transformers/.env/bin/python -m torch.distributed.launch --nproc_per_node=2 /home/github_actions/actions-runner/_work/transformers/transformers/tests/test_trainer_distributed.py --output_dir /tmp/tmp2k265qn5
stderr: Traceback (most recent call last):
stderr:   File &quot;/home/github_actions/actions-runner/_work/transformers/transformers/tests/test_trainer_distributed.py&quot;, line 82, in &lt;module&gt;
stderr:     training_args = parser.parse_args_into_dataclasses()[0]
stderr:   File &quot;/home/github_actions/actions-runner/_work/transformers/transformers/src/transformers/hf_argparser.py&quot;, line 180, in parse_args_into_dataclasses
stderr:     obj = dtype(**inputs)
stderr:   File &quot;&lt;string&gt;&quot;, line 61, in __init__
stderr:   File &quot;/home/github_actions/actions-runner/_work/transformers/transformers/src/transformers/training_args.py&quot;, line 491, in __post_init__
stderr:     if is_torch_available() and self.device.type != &quot;cuda&quot; and self.fp16:
stderr:   File &quot;/home/github_actions/actions-runner/_work/transformers/transformers/src/transformers/file_utils.py&quot;, line 1369, in wrapper
stderr:     return func(*args, **kwargs)
stderr:   File &quot;/home/github_actions/actions-runner/_work/transformers/transformers/src/transformers/training_args.py&quot;, line 620, in device
stderr:     return self._setup_devices
stderr:   File &quot;/home/github_actions/actions-runner/_work/transformers/transformers/src/transformers/file_utils.py&quot;, line 1359, in __get__
stderr:     cached = self.fget(obj)
stderr:   File &quot;/home/github_actions/actions-runner/_work/transformers/transformers/src/transformers/file_utils.py&quot;, line 1369, in wrapper
stderr:     return func(*args, **kwargs)
stderr:   File &quot;/home/github_actions/actions-runner/_work/transformers/transformers/src/transformers/training_args.py&quot;, line 605, in _setup_devices
stderr:     torch.distributed.init_process_group(backend=&quot;nccl&quot;)
stderr:   File &quot;/home/github_actions/actions-runner/_work/transformers/transformers/.env/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py&quot;, line 436, in init_process_group
stderr:     store, rank, world_size = next(rendezvous_iterator)
stderr:   File &quot;/home/github_actions/actions-runner/_work/transformers/transformers/.env/lib/python3.7/site-packages/torch/distributed/rendezvous.py&quot;, line 179, in _env_rendezvous_handler
stderr:     store = TCPStore(master_addr, master_port, world_size, start_daemon, timeout)
stderr: RuntimeError: Address already in use
stderr: Traceback (most recent call last):
stderr:   File &quot;/usr/lib/python3.7/runpy.py&quot;, line 193, in _run_module_as_main
stderr:     &quot;__main__&quot;, mod_spec)
stderr:   File &quot;/usr/lib/python3.7/runpy.py&quot;, line 85, in _run_code
stderr:     exec(code, run_globals)
stderr:   File &quot;/home/github_actions/actions-runner/_work/transformers/transformers/.env/lib/python3.7/site-packages/torch/distributed/launch.py&quot;, line 260, in &lt;module&gt;
stderr:     main()
stderr:   File &quot;/home/github_actions/actions-runner/_work/transformers/transformers/.env/lib/python3.7/site-packages/torch/distributed/launch.py&quot;, line 256, in main
stderr:     cmd=cmd)
stderr: subprocess.CalledProcessError: Command '['/home/github_actions/actions-runner/_work/transformers/transformers/.env/bin/python', '-u', '/home/github_actions/actions-runner/_work/transformers/transformers/tests/test_trainer_distributed.py', '--local_rank=1', '--output_dir', '/tmp/tmp2k265qn5']' returned non-zero exit status 1.
stdout: *****************************************
stdout: Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
stdout: *****************************************
stdout: multi-gpu-ci-runner:18062:18062 [1] NCCL INFO Bootstrap : Using [0]ens6:10.128.0.66&lt;0&gt;
stdout: multi-gpu-ci-runner:18062:18062 [1] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
stdout: 
stdout: multi-gpu-ci-runner:18062:18062 [1] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]
stdout: multi-gpu-ci-runner:18062:18062 [1] NCCL INFO NET/Socket : Using [0]ens6:10.128.0.66&lt;0&gt;
stdout: multi-gpu-ci-runner:18062:18062 [1] NCCL INFO Using network Socket
stdout: multi-gpu-ci-runner:18062:18089 [1] NCCL INFO Call to connect returned Connection refused, retrying
stdout: multi-gpu-ci-runner:18062:18089 [1] NCCL INFO Call to connect returned Connection refused, retrying
stdout: multi-gpu-ci-runner:18062:18089 [1] NCCL INFO Call to connect returned Connection refused, retrying
stdout: multi-gpu-ci-runner:18062:18089 [1] NCCL INFO Call to connect returned Connection refused, retrying
stdout: multi-gpu-ci-runner:18062:18089 [1] NCCL INFO Call to connect returned Connection refused, retrying
stdout: multi-gpu-ci-runner:18062:18089 [1] NCCL INFO Call to connect returned Connection refused, retrying
stdout: multi-gpu-ci-runner:18062:18089 [1] NCCL INFO Call to connect returned Connection refused, retrying
stdout: multi-gpu-ci-runner:18062:18089 [1] NCCL INFO Call to connect returned Connection refused, retrying
stdout: multi-gpu-ci-runner:18062:18089 [1] NCCL INFO Call to connect returned Connection refused, retrying
stdout: multi-gpu-ci-runner:18062:18089 [1] NCCL INFO Call to connect returned Connection refused, retrying
stdout: multi-gpu-ci-runner:18062:18089 [1] NCCL INFO Call to connect returned Connection refused, retrying
stdout: multi-gpu-ci-runner:18062:18089 [1] NCCL INFO Call to connect returned Connection refused, retrying
stdout: multi-gpu-ci-runner:18062:18089 [1] NCCL INFO Call to connect returned Connection refused, retrying
stdout: multi-gpu-ci-runner:18062:18089 [1] NCCL INFO Call to connect returned Connection refused, retrying
stdout: multi-gpu-ci-runner:18062:18089 [1] NCCL INFO Call to connect returned Connection refused, retrying
stdout: multi-gpu-ci-runner:18062:18089 [1] NCCL INFO Call to connect returned Connection refused, retrying
stdout: multi-gpu-ci-runner:18062:18089 [1] NCCL INFO Call to connect returned Connection refused, retrying
stdout: multi-gpu-ci-runner:18062:18089 [1] NCCL INFO Call to connect returned Connection refused, retrying
stdout: multi-gpu-ci-runner:18062:18089 [1] NCCL INFO Call to connect returned Connection refused, retrying
stdout: 
stdout: multi-gpu-ci-runner:18062:18089 [1] include/socket.h:403 NCCL WARN Connect to 10.128.0.66&lt;52523&gt; failed : Connection refused
stdout: multi-gpu-ci-runner:18062:18089 [1] NCCL INFO bootstrap.cc:95 -&gt; 2
stdout: multi-gpu-ci-runner:18062:18089 [1] NCCL INFO bootstrap.cc:309 -&gt; 2
stdout: multi-gpu-ci-runner:18062:18089 [1] NCCL INFO init.cc:555 -&gt; 2
stdout: multi-gpu-ci-runner:18062:18089 [1] NCCL INFO init.cc:840 -&gt; 2
stdout: multi-gpu-ci-runner:18062:18089 [1] NCCL INFO group.cc:73 -&gt; 2 [Async thread]
stderr: Traceback (most recent call last):
stderr:   File &quot;/home/github_actions/actions-runner/_work/transformers/transformers/tests/test_trainer_distributed.py&quot;, line 82, in &lt;module&gt;
stderr:     training_args = parser.parse_args_into_dataclasses()[0]
stderr:   File &quot;/home/github_actions/actions-runner/_work/transformers/transformers/src/transformers/hf_argparser.py&quot;, line 180, in parse_args_into_dataclasses
stderr:     obj = dtype(**inputs)
stderr:   File &quot;&lt;string&gt;&quot;, line 61, in __init__
stderr:   File &quot;/home/github_actions/actions-runner/_work/transformers/transformers/src/transformers/training_args.py&quot;, line 491, in __post_init__
stderr:     if is_torch_available() and self.device.type != &quot;cuda&quot; and self.fp16:
stderr:   File &quot;/home/github_actions/actions-runner/_work/transformers/transformers/src/transformers/file_utils.py&quot;, line 1369, in wrapper
stderr:     return func(*args, **kwargs)
stderr:   File &quot;/home/github_actions/actions-runner/_work/transformers/transformers/src/transformers/training_args.py&quot;, line 620, in device
stderr:     return self._setup_devices
stderr:   File &quot;/home/github_actions/actions-runner/_work/transformers/transformers/src/transformers/file_utils.py&quot;, line 1359, in __get__
stderr:     cached = self.fget(obj)
stderr:   File &quot;/home/github_actions/actions-runner/_work/transformers/transformers/src/transformers/file_utils.py&quot;, line 1369, in wrapper
stderr:     return func(*args, **kwargs)
stderr:   File &quot;/home/github_actions/actions-runner/_work/transformers/transformers/src/transformers/training_args.py&quot;, line 605, in _setup_devices
stderr:     torch.distributed.init_process_group(backend=&quot;nccl&quot;)
stderr:   File &quot;/home/github_actions/actions-runner/_work/transformers/transformers/.env/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py&quot;, line 455, in init_process_group
stderr:     barrier()
stderr:   File &quot;/home/github_actions/actions-runner/_work/transformers/transformers/.env/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py&quot;, line 1960, in barrier
stderr:     work = _default_pg.barrier()
stderr: RuntimeError: NCCL error in: /pytorch/torch/lib/c10d/ProcessGroupNCCL.cpp:784, unhandled system error, NCCL version 2.7.8
FAILED

===================================================================== FAILURES ======================================================================
________________________________________________________ TestTrainerDistributed.test_trainer ________________________________________________________

self = &lt;tests.test_trainer_distributed.TestTrainerDistributed testMethod=test_trainer&gt;

    @require_torch_multi_gpu
    def test_trainer(self):
    
        distributed_args = f&quot;&quot;&quot;
            -m torch.distributed.launch
            --nproc_per_node={torch.cuda.device_count()}
            {self.test_file_dir}/test_trainer_distributed.py
        &quot;&quot;&quot;.split()
        output_dir = self.get_auto_remove_tmp_dir()
        args = f&quot;--output_dir {output_dir}&quot;.split()
        cmd = [sys.executable] + distributed_args + args
&gt;       execute_subprocess_async(cmd, env=self.get_env())

tests/test_trainer_distributed.py:72: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

cmd = ['/home/github_actions/actions-runner/_work/transformers/transformers/.env/bin/python', '-m', 'torch.distributed.launc.../github_actions/actions-runner/_work/transformers/transformers/tests/test_trainer_distributed.py', '--output_dir', ...]
env = {'HOME': '/home/github_actions', 'KMP_DUPLICATE_LIB_OK': 'True', 'KMP_INIT_AT_FORK': 'FALSE', 'LANG': 'C.UTF-8', ...}, stdin = None
timeout = 180, quiet = False, echo = True

    def execute_subprocess_async(cmd, env=None, stdin=None, timeout=180, quiet=False, echo=True) -&gt; _RunOutput:
    
        loop = asyncio.get_event_loop()
        result = loop.run_until_complete(
            _stream_subprocess(cmd, env=env, stdin=stdin, timeout=timeout, quiet=quiet, echo=echo)
        )
    
        cmd_str = &quot; &quot;.join(cmd)
        if result.returncode &gt; 0:
            stderr = &quot;\n&quot;.join(result.stderr)
            raise RuntimeError(
&gt;               f&quot;'{cmd_str}' failed with returncode {result.returncode}\n\n&quot;
                f&quot;The combined stderr from workers follows:\n{stderr}&quot;
            )"><pre class="notranslate"><code class="notranslate">NCCL_DEBUG=INFO pytest -sv tests/test_trainer_distributed.py
================================================================ test session starts ================================================================
platform linux -- Python 3.7.4, pytest-6.2.2, py-1.10.0, pluggy-0.13.1 -- /home/github_actions/actions-runner/_work/transformers/transformers/.env/bin/python
cachedir: .pytest_cache
rootdir: /home/github_actions/actions-runner/_work/transformers/transformers
plugins: xdist-2.2.1, forked-1.3.0
collected 1 item                                                                                                                                    

tests/test_trainer_distributed.py::TestTrainerDistributed::test_trainer 
Running:  /home/github_actions/actions-runner/_work/transformers/transformers/.env/bin/python -m torch.distributed.launch --nproc_per_node=2 /home/github_actions/actions-runner/_work/transformers/transformers/tests/test_trainer_distributed.py --output_dir /tmp/tmp2k265qn5
stderr: Traceback (most recent call last):
stderr:   File "/home/github_actions/actions-runner/_work/transformers/transformers/tests/test_trainer_distributed.py", line 82, in &lt;module&gt;
stderr:     training_args = parser.parse_args_into_dataclasses()[0]
stderr:   File "/home/github_actions/actions-runner/_work/transformers/transformers/src/transformers/hf_argparser.py", line 180, in parse_args_into_dataclasses
stderr:     obj = dtype(**inputs)
stderr:   File "&lt;string&gt;", line 61, in __init__
stderr:   File "/home/github_actions/actions-runner/_work/transformers/transformers/src/transformers/training_args.py", line 491, in __post_init__
stderr:     if is_torch_available() and self.device.type != "cuda" and self.fp16:
stderr:   File "/home/github_actions/actions-runner/_work/transformers/transformers/src/transformers/file_utils.py", line 1369, in wrapper
stderr:     return func(*args, **kwargs)
stderr:   File "/home/github_actions/actions-runner/_work/transformers/transformers/src/transformers/training_args.py", line 620, in device
stderr:     return self._setup_devices
stderr:   File "/home/github_actions/actions-runner/_work/transformers/transformers/src/transformers/file_utils.py", line 1359, in __get__
stderr:     cached = self.fget(obj)
stderr:   File "/home/github_actions/actions-runner/_work/transformers/transformers/src/transformers/file_utils.py", line 1369, in wrapper
stderr:     return func(*args, **kwargs)
stderr:   File "/home/github_actions/actions-runner/_work/transformers/transformers/src/transformers/training_args.py", line 605, in _setup_devices
stderr:     torch.distributed.init_process_group(backend="nccl")
stderr:   File "/home/github_actions/actions-runner/_work/transformers/transformers/.env/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 436, in init_process_group
stderr:     store, rank, world_size = next(rendezvous_iterator)
stderr:   File "/home/github_actions/actions-runner/_work/transformers/transformers/.env/lib/python3.7/site-packages/torch/distributed/rendezvous.py", line 179, in _env_rendezvous_handler
stderr:     store = TCPStore(master_addr, master_port, world_size, start_daemon, timeout)
stderr: RuntimeError: Address already in use
stderr: Traceback (most recent call last):
stderr:   File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
stderr:     "__main__", mod_spec)
stderr:   File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
stderr:     exec(code, run_globals)
stderr:   File "/home/github_actions/actions-runner/_work/transformers/transformers/.env/lib/python3.7/site-packages/torch/distributed/launch.py", line 260, in &lt;module&gt;
stderr:     main()
stderr:   File "/home/github_actions/actions-runner/_work/transformers/transformers/.env/lib/python3.7/site-packages/torch/distributed/launch.py", line 256, in main
stderr:     cmd=cmd)
stderr: subprocess.CalledProcessError: Command '['/home/github_actions/actions-runner/_work/transformers/transformers/.env/bin/python', '-u', '/home/github_actions/actions-runner/_work/transformers/transformers/tests/test_trainer_distributed.py', '--local_rank=1', '--output_dir', '/tmp/tmp2k265qn5']' returned non-zero exit status 1.
stdout: *****************************************
stdout: Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
stdout: *****************************************
stdout: multi-gpu-ci-runner:18062:18062 [1] NCCL INFO Bootstrap : Using [0]ens6:10.128.0.66&lt;0&gt;
stdout: multi-gpu-ci-runner:18062:18062 [1] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
stdout: 
stdout: multi-gpu-ci-runner:18062:18062 [1] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]
stdout: multi-gpu-ci-runner:18062:18062 [1] NCCL INFO NET/Socket : Using [0]ens6:10.128.0.66&lt;0&gt;
stdout: multi-gpu-ci-runner:18062:18062 [1] NCCL INFO Using network Socket
stdout: multi-gpu-ci-runner:18062:18089 [1] NCCL INFO Call to connect returned Connection refused, retrying
stdout: multi-gpu-ci-runner:18062:18089 [1] NCCL INFO Call to connect returned Connection refused, retrying
stdout: multi-gpu-ci-runner:18062:18089 [1] NCCL INFO Call to connect returned Connection refused, retrying
stdout: multi-gpu-ci-runner:18062:18089 [1] NCCL INFO Call to connect returned Connection refused, retrying
stdout: multi-gpu-ci-runner:18062:18089 [1] NCCL INFO Call to connect returned Connection refused, retrying
stdout: multi-gpu-ci-runner:18062:18089 [1] NCCL INFO Call to connect returned Connection refused, retrying
stdout: multi-gpu-ci-runner:18062:18089 [1] NCCL INFO Call to connect returned Connection refused, retrying
stdout: multi-gpu-ci-runner:18062:18089 [1] NCCL INFO Call to connect returned Connection refused, retrying
stdout: multi-gpu-ci-runner:18062:18089 [1] NCCL INFO Call to connect returned Connection refused, retrying
stdout: multi-gpu-ci-runner:18062:18089 [1] NCCL INFO Call to connect returned Connection refused, retrying
stdout: multi-gpu-ci-runner:18062:18089 [1] NCCL INFO Call to connect returned Connection refused, retrying
stdout: multi-gpu-ci-runner:18062:18089 [1] NCCL INFO Call to connect returned Connection refused, retrying
stdout: multi-gpu-ci-runner:18062:18089 [1] NCCL INFO Call to connect returned Connection refused, retrying
stdout: multi-gpu-ci-runner:18062:18089 [1] NCCL INFO Call to connect returned Connection refused, retrying
stdout: multi-gpu-ci-runner:18062:18089 [1] NCCL INFO Call to connect returned Connection refused, retrying
stdout: multi-gpu-ci-runner:18062:18089 [1] NCCL INFO Call to connect returned Connection refused, retrying
stdout: multi-gpu-ci-runner:18062:18089 [1] NCCL INFO Call to connect returned Connection refused, retrying
stdout: multi-gpu-ci-runner:18062:18089 [1] NCCL INFO Call to connect returned Connection refused, retrying
stdout: multi-gpu-ci-runner:18062:18089 [1] NCCL INFO Call to connect returned Connection refused, retrying
stdout: 
stdout: multi-gpu-ci-runner:18062:18089 [1] include/socket.h:403 NCCL WARN Connect to 10.128.0.66&lt;52523&gt; failed : Connection refused
stdout: multi-gpu-ci-runner:18062:18089 [1] NCCL INFO bootstrap.cc:95 -&gt; 2
stdout: multi-gpu-ci-runner:18062:18089 [1] NCCL INFO bootstrap.cc:309 -&gt; 2
stdout: multi-gpu-ci-runner:18062:18089 [1] NCCL INFO init.cc:555 -&gt; 2
stdout: multi-gpu-ci-runner:18062:18089 [1] NCCL INFO init.cc:840 -&gt; 2
stdout: multi-gpu-ci-runner:18062:18089 [1] NCCL INFO group.cc:73 -&gt; 2 [Async thread]
stderr: Traceback (most recent call last):
stderr:   File "/home/github_actions/actions-runner/_work/transformers/transformers/tests/test_trainer_distributed.py", line 82, in &lt;module&gt;
stderr:     training_args = parser.parse_args_into_dataclasses()[0]
stderr:   File "/home/github_actions/actions-runner/_work/transformers/transformers/src/transformers/hf_argparser.py", line 180, in parse_args_into_dataclasses
stderr:     obj = dtype(**inputs)
stderr:   File "&lt;string&gt;", line 61, in __init__
stderr:   File "/home/github_actions/actions-runner/_work/transformers/transformers/src/transformers/training_args.py", line 491, in __post_init__
stderr:     if is_torch_available() and self.device.type != "cuda" and self.fp16:
stderr:   File "/home/github_actions/actions-runner/_work/transformers/transformers/src/transformers/file_utils.py", line 1369, in wrapper
stderr:     return func(*args, **kwargs)
stderr:   File "/home/github_actions/actions-runner/_work/transformers/transformers/src/transformers/training_args.py", line 620, in device
stderr:     return self._setup_devices
stderr:   File "/home/github_actions/actions-runner/_work/transformers/transformers/src/transformers/file_utils.py", line 1359, in __get__
stderr:     cached = self.fget(obj)
stderr:   File "/home/github_actions/actions-runner/_work/transformers/transformers/src/transformers/file_utils.py", line 1369, in wrapper
stderr:     return func(*args, **kwargs)
stderr:   File "/home/github_actions/actions-runner/_work/transformers/transformers/src/transformers/training_args.py", line 605, in _setup_devices
stderr:     torch.distributed.init_process_group(backend="nccl")
stderr:   File "/home/github_actions/actions-runner/_work/transformers/transformers/.env/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 455, in init_process_group
stderr:     barrier()
stderr:   File "/home/github_actions/actions-runner/_work/transformers/transformers/.env/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 1960, in barrier
stderr:     work = _default_pg.barrier()
stderr: RuntimeError: NCCL error in: /pytorch/torch/lib/c10d/ProcessGroupNCCL.cpp:784, unhandled system error, NCCL version 2.7.8
FAILED

===================================================================== FAILURES ======================================================================
________________________________________________________ TestTrainerDistributed.test_trainer ________________________________________________________

self = &lt;tests.test_trainer_distributed.TestTrainerDistributed testMethod=test_trainer&gt;

    @require_torch_multi_gpu
    def test_trainer(self):
    
        distributed_args = f"""
            -m torch.distributed.launch
            --nproc_per_node={torch.cuda.device_count()}
            {self.test_file_dir}/test_trainer_distributed.py
        """.split()
        output_dir = self.get_auto_remove_tmp_dir()
        args = f"--output_dir {output_dir}".split()
        cmd = [sys.executable] + distributed_args + args
&gt;       execute_subprocess_async(cmd, env=self.get_env())

tests/test_trainer_distributed.py:72: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

cmd = ['/home/github_actions/actions-runner/_work/transformers/transformers/.env/bin/python', '-m', 'torch.distributed.launc.../github_actions/actions-runner/_work/transformers/transformers/tests/test_trainer_distributed.py', '--output_dir', ...]
env = {'HOME': '/home/github_actions', 'KMP_DUPLICATE_LIB_OK': 'True', 'KMP_INIT_AT_FORK': 'FALSE', 'LANG': 'C.UTF-8', ...}, stdin = None
timeout = 180, quiet = False, echo = True

    def execute_subprocess_async(cmd, env=None, stdin=None, timeout=180, quiet=False, echo=True) -&gt; _RunOutput:
    
        loop = asyncio.get_event_loop()
        result = loop.run_until_complete(
            _stream_subprocess(cmd, env=env, stdin=stdin, timeout=timeout, quiet=quiet, echo=echo)
        )
    
        cmd_str = " ".join(cmd)
        if result.returncode &gt; 0:
            stderr = "\n".join(result.stderr)
            raise RuntimeError(
&gt;               f"'{cmd_str}' failed with returncode {result.returncode}\n\n"
                f"The combined stderr from workers follows:\n{stderr}"
            )
</code></pre></div>
<p dir="auto">A short term workaround could be to randomize the port, so this test won't trumple upon its previous zombie.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="+        from random import randint
+        master_port = 2950 + randint(1, 99)
         distributed_args = f&quot;&quot;&quot;
             -m torch.distributed.launch
             --nproc_per_node={torch.cuda.device_count()}
+            --master_port {master_port}
             {self.test_file_dir}/test_trainer_distributed.py
         &quot;&quot;&quot;.split()"><pre class="notranslate"><code class="notranslate">+        from random import randint
+        master_port = 2950 + randint(1, 99)
         distributed_args = f"""
             -m torch.distributed.launch
             --nproc_per_node={torch.cuda.device_count()}
+            --master_port {master_port}
             {self.test_file_dir}/test_trainer_distributed.py
         """.split()
</code></pre></div>
<p dir="auto">but this is a band-aid and a real solution is needed. It also will be an issue with any other distributed tests that rely on the same default port number.</p>
<p dir="auto">I will keep on monitoring the issue.</p>
<p dir="auto">Meanwhile this PR <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="812198366" data-permission-text="Title is private" data-url="https://github.com/huggingface/transformers/issues/10281" data-hovercard-type="pull_request" data-hovercard-url="/huggingface/transformers/pull/10281/hovercard" href="https://github.com/huggingface/transformers/pull/10281">#10281</a> should help with preventing the incremental number of zombies from scheduled runs.</p>
<p dir="auto">It's difficult to debug w/o being able to reproduce this problem at will.</p>