<h2 dir="auto">I submitted a 4-node task with 1GPU for each node.<br>
But it exit with exception.<br>
Some of the log information is as follows:<br>
NCCL WARN Connect to 10.38.10.112&lt;21724&gt; failed : Connection refused<br>
The strange thing is that none of the 4 nodes's ip is 10.38.10.112&lt;21724&gt;. I don't know why it will try to connect the ip and the port .<br>
Besides,<br>
I have set the NCCL_SOCKET_IFNAME to "^lo,docker", I used ifconfig to check network config of all the nodes. And I am sure that configs are  'eth0, eth1, and lo".<br>
One of the node info is given as below:<br>
self.dist_backend: nccl<br>
self.dist_init_method: file:///home/storage15/huangying/tools/espnet/egs2/voxforge/asr1/vox.init<br>
self.dist_world_size: 4<br>
self.dist_rank: 1<br>
auto allocate gpu device: 0<br>
devices ids is  0</h2>
<p dir="auto">tj1-asr-train-v100-13:941227:943077 [0] NCCL INFO Call to connect returned Connection refused, retrying<br>
tj1-asr-train-v100-13:941227:943077 [0] NCCL INFO Call to connect returned Connection refused, retrying<br>
tj1-asr-train-v100-13:941227:943077 [0] NCCL INFO Call to connect returned Connection refused, retrying</p>
<p dir="auto">tj1-asr-train-v100-13:941227:943077 [0] include/socket.h:390 NCCL WARN Connect to 10.38.10.112&lt;21724&gt; failed : Connection refused<br>
tj1-asr-train-v100-13:941227:943077 [0] NCCL INFO bootstrap.cc:100 -&gt; 2<br>
tj1-asr-train-v100-13:941227:943077 [0] NCCL INFO bootstrap.cc:326 -&gt; 2<br>
tj1-asr-train-v100-13:941227:943077 [0] NCCL INFO init.cc:695 -&gt; 2<br>
tj1-asr-train-v100-13:941227:943077 [0] NCCL INFO init.cc:951 -&gt; 2<br>
tj1-asr-train-v100-13:941227:943077 [0] NCCL INFO misc/group.cc:69 -&gt; 2 [Async thread]<br>
/home/storage15/huangying/tools/anaconda3/envs/py36/lib/python3.6/site-packages/librosa/util/decorators.py:9: NumbaDeprecationWarning: An import was requested from a module that has moved location.<br>
Import of 'jit' requested from: 'numba.decorators', please update to use 'numba.core.decorators' or pin to Numba version 0.48.0. This alias will not be present in Numba version 0.50.0.<br>
from numba.decorators import jit as optional_jit<br>
/home/storage15/huangying/tools/anaconda3/envs/py36/bin/python3 /home/storage15/huangying/tools/espnet/espnet2/bin/asr_train.py --use_preprocessor true --bpemodel none --token_type char --token_list data/token_list/char/tokens.txt --non_linguistic_symbols none --train_data_path_and_name_and_type dump/fbank_pitch/tr_en/feats.scp,speech,kaldi_ark --train_data_path_and_name_and_type dump/fbank_pitch/tr_en/text,text,text --valid_data_path_and_name_and_type dump/fbank_pitch/dt_en/feats.scp,speech,kaldi_ark --valid_data_path_and_name_and_type dump/fbank_pitch/dt_en/text,text,text --train_shape_file exp/asr_stats/train/speech_shape --train_shape_file exp/asr_stats/train/text_shape.char --valid_shape_file exp/asr_stats/valid/speech_shape --valid_shape_file exp/asr_stats/valid/text_shape.char --resume true --fold_length 800 --fold_length 150 --output_dir exp/asr_train_asr_transformer_fbank_pitch_char_normalize_confnorm_varsFalse --ngpu 1 --dist_init_method file:///home/storage15/huangying/tools/espnet/egs2/voxforge/asr1/vox.init --multiprocessing_distributed false --dist_launcher queue.pl --dist_world_size 4 --config conf/train_asr_transformer.yaml --input_size=83 --normalize=global_mvn --normalize_conf stats_file=exp/asr_stats/train/feats_stats.npz --normalize_conf norm_vars=False<br>
Traceback (most recent call last):<br>
File "/home/storage15/huangying/tools/anaconda3/envs/py36/lib/python3.6/runpy.py", line 193, in _run_module_as_main<br>
"<strong>main</strong>", mod_spec)<br>
File "/home/storage15/huangying/tools/anaconda3/envs/py36/lib/python3.6/runpy.py", line 85, in _run_code<br>
exec(code, run_globals)<br>
File "/home/storage15/huangying/tools/espnet/espnet2/bin/asr_train.py", line 23, in <br>
main()<br>
File "/home/storage15/huangying/tools/espnet/espnet2/bin/asr_train.py", line 19, in main<br>
ASRTask.main(cmd=cmd)<br>
File "/home/storage15/huangying/tools/espnet/espnet2/tasks/abs_task.py", line 842, in main<br>
cls.main_worker(args)<br>
File "/home/storage15/huangying/tools/espnet/espnet2/tasks/abs_task.py", line 1174, in main_worker<br>
distributed_option=distributed_option,<br>
File "/home/storage15/huangying/tools/espnet/espnet2/train/trainer.py", line 163, in run<br>
else None<br>
File "/home/storage15/huangying/tools/anaconda3/envs/py36/lib/python3.6/site-packages/torch/nn/parallel/distributed.py", line 303, in <strong>init</strong><br>
self.broadcast_bucket_size)<br>
File "/home/storage15/huangying/tools/anaconda3/envs/py36/lib/python3.6/site-packages/torch/nn/parallel/distributed.py", line 485, in _distributed_broadcast_coalesced<br>
dist._broadcast_coalesced(self.process_group, tensors, buffer_size)<br>
RuntimeError: NCCL error in: /pytorch/torch/lib/c10d/ProcessGroupNCCL.cpp:410, unhandled system error, NCCL version 2.4.8<br>
Setting NCCL_SOCKET_IFNAME<br>
Finished NCCL_SOCKET_IFNAME<br>
^lo,docker<br>
self.dist_backend: nccl<br>
self.dist_init_method: file:///home/storage15/huangying/tools/espnet/egs2/voxforge/asr1/vox.init<br>
self.dist_world_size: 4<br>
self.dist_rank: 1<br>
auto allocate gpu device: 0<br>
devices ids is  0</p>
<h1 dir="auto">Accounting: time=107 threads=1</h1>
<h1 dir="auto">Finished at Mon May 18 14:45:28 CST 2020 with status 1</h1>
<p dir="auto">cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/pietern/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/pietern">@pietern</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/mrshenli/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/mrshenli">@mrshenli</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/pritamdamania87/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/pritamdamania87">@pritamdamania87</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/zhaojuanmao/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/zhaojuanmao">@zhaojuanmao</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/satgera/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/satgera">@satgera</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/rohan-varma/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/rohan-varma">@rohan-varma</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/gqchen/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/gqchen">@gqchen</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/aazzolini/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/aazzolini">@aazzolini</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/xush6528/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/xush6528">@xush6528</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/osalpekar/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/osalpekar">@osalpekar</a></p>
<h2 dir="auto"><g-emoji class="g-emoji" alias="bug" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f41b.png">🐛</g-emoji> Bug</h2>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="  for j in range(seqlen):
Traceback (most recent call last):
  File &quot;onnx_full_export.py&quot;, line 74, in &lt;module&gt;
    main()
  File &quot;onnx_full_export.py&quot;, line 70, in main
    beam_search.save_to_db(args.output_file)
  File &quot;/home/raden/translate/pytorch_translate/ensemble_export.py&quot;, line 1170, in save_to_db
    self.onnx_export(tmp_file)
  File &quot;/home/raden/translate/pytorch_translate/ensemble_export.py&quot;, line 1130, in onnx_export
    export_type=ExportTypes.ZIP_ARCHIVE,
  File &quot;/home/raden/anaconda3/lib/python3.7/site-packages/torch/onnx/__init__.py&quot;, line 19, in _export
    result = utils._export(*args, **kwargs)
  File &quot;/home/raden/anaconda3/lib/python3.7/site-packages/torch/onnx/utils.py&quot;, line 363, in _export
    _retain_param_name, do_constant_folding)
  File &quot;/home/raden/anaconda3/lib/python3.7/site-packages/torch/onnx/utils.py&quot;, line 256, in _model_to_graph
    method.graph, tuple(args) + tuple(params), example_outputs, False, propagate)
RuntimeError: retval-&gt;inputs().size() == inputs.size() ASSERT FAILED at /pytorch/torch/csrc/jit/script/init.cpp:744, please report a bug to PyTorch. (_propagate_and_assign_input_and_output_shapes at /pytorch/torch/csrc/jit/script/init.cpp:744)
frame #0: std::function&lt;std::string ()&gt;::operator()() const + 0x11 (0x7f6b9c93b441 in /home/raden/anaconda3/lib/python3.7/site-packages/torch/lib/libc10.so)
frame #1: c10::Error::Error(c10::SourceLocation, std::string const&amp;) + 0x2a (0x7f6b9c93ad7a in /home/raden/anaconda3/lib/python3.7/site-packages/torch/lib/libc10.so)
frame #2: &lt;unknown function&gt; + 0x452376 (0x7f6bdbf5c376 in /home/raden/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #3: &lt;unknown function&gt; + 0x4793af (0x7f6bdbf833af in /home/raden/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #4: &lt;unknown function&gt; + 0x130fac (0x7f6bdbc3afac in /home/raden/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
&lt;omitting python frames&gt;
frame #31: __libc_start_main + 0xf0 (0x7f6beb660830 in /lib/x86_64-linux-gnu/libc.so.6)

"><pre class="notranslate"><code class="notranslate">  for j in range(seqlen):
Traceback (most recent call last):
  File "onnx_full_export.py", line 74, in &lt;module&gt;
    main()
  File "onnx_full_export.py", line 70, in main
    beam_search.save_to_db(args.output_file)
  File "/home/raden/translate/pytorch_translate/ensemble_export.py", line 1170, in save_to_db
    self.onnx_export(tmp_file)
  File "/home/raden/translate/pytorch_translate/ensemble_export.py", line 1130, in onnx_export
    export_type=ExportTypes.ZIP_ARCHIVE,
  File "/home/raden/anaconda3/lib/python3.7/site-packages/torch/onnx/__init__.py", line 19, in _export
    result = utils._export(*args, **kwargs)
  File "/home/raden/anaconda3/lib/python3.7/site-packages/torch/onnx/utils.py", line 363, in _export
    _retain_param_name, do_constant_folding)
  File "/home/raden/anaconda3/lib/python3.7/site-packages/torch/onnx/utils.py", line 256, in _model_to_graph
    method.graph, tuple(args) + tuple(params), example_outputs, False, propagate)
RuntimeError: retval-&gt;inputs().size() == inputs.size() ASSERT FAILED at /pytorch/torch/csrc/jit/script/init.cpp:744, please report a bug to PyTorch. (_propagate_and_assign_input_and_output_shapes at /pytorch/torch/csrc/jit/script/init.cpp:744)
frame #0: std::function&lt;std::string ()&gt;::operator()() const + 0x11 (0x7f6b9c93b441 in /home/raden/anaconda3/lib/python3.7/site-packages/torch/lib/libc10.so)
frame #1: c10::Error::Error(c10::SourceLocation, std::string const&amp;) + 0x2a (0x7f6b9c93ad7a in /home/raden/anaconda3/lib/python3.7/site-packages/torch/lib/libc10.so)
frame #2: &lt;unknown function&gt; + 0x452376 (0x7f6bdbf5c376 in /home/raden/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #3: &lt;unknown function&gt; + 0x4793af (0x7f6bdbf833af in /home/raden/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #4: &lt;unknown function&gt; + 0x130fac (0x7f6bdbc3afac in /home/raden/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
&lt;omitting python frames&gt;
frame #31: __libc_start_main + 0xf0 (0x7f6beb660830 in /lib/x86_64-linux-gnu/libc.so.6)


</code></pre></div>
<h2 dir="auto">To Reproduce</h2>
<p dir="auto">Steps to reproduce the behavior:</p>
<p dir="auto">Run onnx_full_export.py from <a href="https://github.com/pytorch/translate">https://github.com/pytorch/translate</a></p>

<h2 dir="auto">Expected behavior</h2>

<h2 dir="auto">Environment</h2>
<p dir="auto">Pytorch 1.1 stable</p>
<h2 dir="auto">Additional context</h2>