<p dir="auto">Hi, When I build dockerfile, I see error message about git submodule.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Step 10/14 : RUN git submodule update --init
 ---&gt; Running in 0633cd9d62c6
fatal: Not a git repository (or any of the parent directories): .git
The command '/bin/sh -c git submodule update --init' returned a non-zero code: 128"><pre class="notranslate"><code class="notranslate">Step 10/14 : RUN git submodule update --init
 ---&gt; Running in 0633cd9d62c6
fatal: Not a git repository (or any of the parent directories): .git
The command '/bin/sh -c git submodule update --init' returned a non-zero code: 128
</code></pre></div>
<p dir="auto">Ubuntu16.04 and git version 2.16.1.</p>
<h2 dir="auto"><g-emoji class="g-emoji" alias="bug" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f41b.png">🐛</g-emoji> Bug</h2>
<p dir="auto">PR <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="672516878" data-permission-text="Title is private" data-url="https://github.com/pytorch/pytorch/issues/42514" data-hovercard-type="pull_request" data-hovercard-url="/pytorch/pytorch/pull/42514/hovercard" href="https://github.com/pytorch/pytorch/pull/42514">#42514</a> added NCCL Alltoall support to NCCL process group using NCCL Send/Recv. This seems to cause unrelated NCCL tests to time-out. This feature has been disabled by default.</p>
<h2 dir="auto">To Reproduce</h2>
<p dir="auto">Steps to reproduce the behavior:</p>
<p dir="auto">Set ENABLE_NCCL_A2A=1 to enable NCCL A2A support using NCCL send/recv. See: <a href="https://github.com/pytorch/pytorch/pull/42514/files#diff-5e6ecb5ae373ed102c2d012ea8847336R21">https://github.com/pytorch/pytorch/pull/42514/files#diff-5e6ecb5ae373ed102c2d012ea8847336R21</a></p>
<p dir="auto">This will cause few/all of the following jobs to time-out:<br>
pytorch_linux_xenial_cuda10_2_cudnn7_py3_ge_config_profiling_test<br>
pytorch_linux_xenial_cuda10_2_cudnn7_py3_ge_config_legacy_test<br>
pytorch_linux_xenial_cuda10_2_cudnn7_py3_gcc7_test</p>
<p dir="auto">This happens even if the test_all_to_all_single_equal_split_cuda and other NCCL alltoall tests are not enabled:<br>
<a href="https://github.com/pytorch/pytorch/pull/42514/files#diff-3469d9b41904e6ed24347065234034e2R1575">https://github.com/pytorch/pytorch/pull/42514/files#diff-3469d9b41904e6ed24347065234034e2R1575</a><br>
<a href="https://github.com/pytorch/pytorch/pull/42514/files#diff-3469d9b41904e6ed24347065234034e2R1599">https://github.com/pytorch/pytorch/pull/42514/files#diff-3469d9b41904e6ed24347065234034e2R1599</a><br>
<a href="https://github.com/pytorch/pytorch/pull/42514/files#diff-3469d9b41904e6ed24347065234034e2R1599">https://github.com/pytorch/pytorch/pull/42514/files#diff-3469d9b41904e6ed24347065234034e2R1599</a><br>
<a href="https://github.com/pytorch/pytorch/pull/42514/files#diff-3469d9b41904e6ed24347065234034e2R1599">https://github.com/pytorch/pytorch/pull/42514/files#diff-3469d9b41904e6ed24347065234034e2R1599</a></p>
<p dir="auto">The issue can be reproduced by ssh'ing into one of the containers for the above jobs and running:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="python test_nn.py --verbose TestNN.test_broadcast_double_backwards_gpu 

    frame #1: 0x00007fffac982755 libtorch_cuda.so`ncclBarrierEnqueueWait(ncclComm*) + 43 at enqueue.cc:162
    frame #2: 0x00007fffac98272a libtorch_cuda.so`ncclBarrierEnqueueWait(comm=0x00007fff10000e00) + 218 at enqueue.cc:217
    frame #3: 0x00007fffac98345e libtorch_cuda.so`ncclEnqueueCheck(info=0x00007fffffff9510) + 1998 at enqueue.cc:566
    frame #4: 0x00007fffac999ce9 libtorch_cuda.so`::ncclBroadcast(sendbuff=&lt;unavailable&gt;, recvbuff=&lt;unavailable&gt;, count=&lt;unavailable&gt;, datatype=&lt;unavailable&gt;, root=&lt;unavailable&gt;, comm=&lt;unavailable&gt;, stream=0x000055555775eea0) + 121 at broadcast.cc:17
    frame #5: 0x00007fffac999d29 libtorch_cuda.so`::ncclBcast(buff=&lt;unavailable&gt;, count=&lt;unavailable&gt;, datatype=&lt;unavailable&gt;, root=&lt;unavailable&gt;, comm=&lt;unavailable&gt;, stream=&lt;unavailable&gt;) + 25 at broadcast.cc:24
    frame #6: 0x00007fffaa8490ee libtorch_cuda.so`torch::cuda::nccl::broadcast(c10::ArrayRef&lt;at::Tensor&gt;, std::vector&lt;c10::optional&lt;c10::cuda::CUDAStream&gt;, std::allocator&lt;c10::optional&lt;c10::cuda::CUDAStream&gt; &gt; &gt; const&amp;, std::vector&lt;ncclComm*, std::allocator&lt;ncclComm*&gt; &gt; const&amp;) + 622
    frame #7: 0x00007fffaa7067fd libtorch_cuda.so`torch::cuda::_broadcast_out_impl(at::Tensor const&amp;, std::vector&lt;at::Tensor, std::allocator&lt;at::Tensor&gt; &gt;&amp;) + 445
    frame #8: 0x00007fffaa706c4f libtorch_cuda.so`torch::cuda::broadcast(at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;) + 975
    frame #9: 0x00007fffaa707a4a libtorch_cuda.so`torch::cuda::broadcast_coalesced(c10::ArrayRef&lt;at::Tensor&gt;, c10::ArrayRef&lt;long&gt;, unsigned long) + 2746"><pre class="notranslate"><code class="notranslate">python test_nn.py --verbose TestNN.test_broadcast_double_backwards_gpu 

    frame #1: 0x00007fffac982755 libtorch_cuda.so`ncclBarrierEnqueueWait(ncclComm*) + 43 at enqueue.cc:162
    frame #2: 0x00007fffac98272a libtorch_cuda.so`ncclBarrierEnqueueWait(comm=0x00007fff10000e00) + 218 at enqueue.cc:217
    frame #3: 0x00007fffac98345e libtorch_cuda.so`ncclEnqueueCheck(info=0x00007fffffff9510) + 1998 at enqueue.cc:566
    frame #4: 0x00007fffac999ce9 libtorch_cuda.so`::ncclBroadcast(sendbuff=&lt;unavailable&gt;, recvbuff=&lt;unavailable&gt;, count=&lt;unavailable&gt;, datatype=&lt;unavailable&gt;, root=&lt;unavailable&gt;, comm=&lt;unavailable&gt;, stream=0x000055555775eea0) + 121 at broadcast.cc:17
    frame #5: 0x00007fffac999d29 libtorch_cuda.so`::ncclBcast(buff=&lt;unavailable&gt;, count=&lt;unavailable&gt;, datatype=&lt;unavailable&gt;, root=&lt;unavailable&gt;, comm=&lt;unavailable&gt;, stream=&lt;unavailable&gt;) + 25 at broadcast.cc:24
    frame #6: 0x00007fffaa8490ee libtorch_cuda.so`torch::cuda::nccl::broadcast(c10::ArrayRef&lt;at::Tensor&gt;, std::vector&lt;c10::optional&lt;c10::cuda::CUDAStream&gt;, std::allocator&lt;c10::optional&lt;c10::cuda::CUDAStream&gt; &gt; &gt; const&amp;, std::vector&lt;ncclComm*, std::allocator&lt;ncclComm*&gt; &gt; const&amp;) + 622
    frame #7: 0x00007fffaa7067fd libtorch_cuda.so`torch::cuda::_broadcast_out_impl(at::Tensor const&amp;, std::vector&lt;at::Tensor, std::allocator&lt;at::Tensor&gt; &gt;&amp;) + 445
    frame #8: 0x00007fffaa706c4f libtorch_cuda.so`torch::cuda::broadcast(at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;) + 975
    frame #9: 0x00007fffaa707a4a libtorch_cuda.so`torch::cuda::broadcast_coalesced(c10::ArrayRef&lt;at::Tensor&gt;, c10::ArrayRef&lt;long&gt;, unsigned long) + 2746
</code></pre></div>
<p dir="auto">In other words, it appears simply adding NCCL a2a code using NCCL send/recv calls (without any tests calling NCCL a2a) seems to trigger the issue. For instance, the issue has been isolated to <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="669258815" data-permission-text="Title is private" data-url="https://github.com/pytorch/pytorch/issues/42339" data-hovercard-type="pull_request" data-hovercard-url="/pytorch/pytorch/pull/42339/hovercard" href="https://github.com/pytorch/pytorch/pull/42339">#42339</a> which adds the code but does not call it from tests.<br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="669261695" data-permission-text="Title is private" data-url="https://github.com/pytorch/pytorch/issues/42340" data-hovercard-type="pull_request" data-hovercard-url="/pytorch/pytorch/pull/42340/hovercard" href="https://github.com/pytorch/pytorch/pull/42340">#42340</a> [Don't Review] Adding one all2all test<br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="669258815" data-permission-text="Title is private" data-url="https://github.com/pytorch/pytorch/issues/42339" data-hovercard-type="pull_request" data-hovercard-url="/pytorch/pytorch/pull/42339/hovercard" href="https://github.com/pytorch/pytorch/pull/42339">#42339</a> [Don't Review] Adding ProcessGroupNCCL::alltoall_base<br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="669257444" data-permission-text="Title is private" data-url="https://github.com/pytorch/pytorch/issues/42338" data-hovercard-type="pull_request" data-hovercard-url="/pytorch/pytorch/pull/42338/hovercard" href="https://github.com/pytorch/pytorch/pull/42338">#42338</a> [Don't Review] Adding ProcessGroupNCCL::alltoall<br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="669254456" data-permission-text="Title is private" data-url="https://github.com/pytorch/pytorch/issues/42337" data-hovercard-type="pull_request" data-hovercard-url="/pytorch/pytorch/pull/42337/hovercard" href="https://github.com/pytorch/pytorch/pull/42337">#42337</a> [Don't Review] Move all2all helper functions to base class</p>
<h2 dir="auto">Expected behavior</h2>
<p dir="auto">Given NCCL 2.7.0 support, it is expected NCCL Alltoall support and the respective tests in test_distributed.py to execute without any issues. This has been confirmed in stand-alone environments like DGX-1.</p>
<h2 dir="auto">Environment</h2>
<p dir="auto">This issue seems to specifically arise in Github/CircleCI test environment.</p>
<p dir="auto">cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/malfet/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/malfet">@malfet</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/seemethere/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/seemethere">@seemethere</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/walterddr/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/walterddr">@walterddr</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ngimel/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ngimel">@ngimel</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/pietern/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/pietern">@pietern</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/mrshenli/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/mrshenli">@mrshenli</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/pritamdamania87/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/pritamdamania87">@pritamdamania87</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/zhaojuanmao/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/zhaojuanmao">@zhaojuanmao</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/satgera/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/satgera">@satgera</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/rohan-varma/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/rohan-varma">@rohan-varma</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/gqchen/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/gqchen">@gqchen</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/aazzolini/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/aazzolini">@aazzolini</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/xush6528/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/xush6528">@xush6528</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/osalpekar/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/osalpekar">@osalpekar</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/jiayisuse/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/jiayisuse">@jiayisuse</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/agolynski/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/agolynski">@agolynski</a></p>