<h1 dir="auto">Description</h1>
<p dir="auto">I've tried finding resources regarding to this but they seem outdated.<br>
The problem is that I am trying to build in visual studio with the binaries I downloaded from</p>
<p dir="auto"><code class="notranslate">https://download.pytorch.org/libtorch/cu102/libtorch-win-shared-with-deps-debug-1.5.0.zip</code></p>
<p dir="auto">I have set the following properties in Visual Studio 2019</p>
<ol dir="auto">
<li>Additional include directories : <code class="notranslate">libtorch\include</code> ; <code class="notranslate">libtorch\include\torch\csrc\api\include\</code></li>
<li>Additional Library Directory for linker : <code class="notranslate">\libtorch\lib\</code></li>
<li>Conformance mode to No</li>
<li>Linker input : <code class="notranslate">torch.lib;c10.lib</code><br>
However the resources are also suggesting libs like <code class="notranslate">caffe2_nvrtc.lib</code> and <code class="notranslate">caffe2.lib</code> but they are not present in <code class="notranslate">\libtorch\lib\</code></li>
</ol>
<p dir="auto">I tried to run the following program</p>
<div class="highlight highlight-source-c++ notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="#include&lt;torch/torch.h&gt;
#include&lt;iostream&gt;
int main()
{
	torch::Tensor tensor = torch::eye(3);
	std::cout &lt;&lt; tensor &lt;&lt; std::endl;
}"><pre class="notranslate">#<span class="pl-k">include</span><span class="pl-s"><span class="pl-pds">&lt;</span>torch/torch.h<span class="pl-pds">&gt;</span></span>
#<span class="pl-k">include</span><span class="pl-s"><span class="pl-pds">&lt;</span>iostream<span class="pl-pds">&gt;</span></span>
<span class="pl-k">int</span> <span class="pl-en">main</span>()
{
	torch::Tensor tensor = <span class="pl-c1">torch::eye</span>(<span class="pl-c1">3</span>);
	std::cout &lt;&lt; tensor &lt;&lt; std::endl;
}</pre></div>
<p dir="auto">I got series of errors and warnings :</p>
<p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://user-images.githubusercontent.com/49276301/83049178-646b0680-a068-11ea-85d8-af5c58cac6f4.png"><img src="https://user-images.githubusercontent.com/49276301/83049178-646b0680-a068-11ea-85d8-af5c58cac6f4.png" alt="Screenshot (48)" style="max-width: 100%;"></a></p>
<p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://user-images.githubusercontent.com/49276301/83049521-f115c480-a068-11ea-824f-c1d4cb633131.png"><img src="https://user-images.githubusercontent.com/49276301/83049521-f115c480-a068-11ea-824f-c1d4cb633131.png" alt="Screenshot (49)" style="max-width: 100%;"></a></p>
<p dir="auto">Operating System :  Windows 10</p>
<h2 dir="auto"><g-emoji class="g-emoji" alias="bug" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f41b.png">🐛</g-emoji> Bug</h2>
<ol dir="auto">
<li>Running <code class="notranslate">TestDistBackend.test_DistributedDataParallel</code> from <code class="notranslate">test_distributed</code> suite</li>
<li>with a WORLD_SIZE &lt;= nGPUs/2 on the machine<br>
results in an error "Model replicas must have an equal number of parameters".</li>
</ol>
<h2 dir="auto">To Reproduce</h2>
<p dir="auto">Steps to reproduce the behavior:</p>
<ol dir="auto">
<li>Use a machine which has at least 4 GPUs</li>
<li>Build pytorch from source for python3.6 OR use one of the available docker images.</li>
<li>Run the following command: <code class="notranslate">BACKEND=nccl WORLD_SIZE=2 TEMP_DIR=/tmp python3.6 test_distributed.py --verbose TestDistBackend.test_DistributedDataParallel</code></li>
</ol>
<p dir="auto">On a ROCm build of PyTorch, I get the below stack trace (although this issue isn't ROCm-specific):</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Process process 0:
Traceback (most recent call last):
  File &quot;/opt/conda/lib/python3.6/multiprocessing/process.py&quot;, line 258, in _bootstrap
    self.run()
  File &quot;/opt/conda/lib/python3.6/multiprocessing/process.py&quot;, line 93, in run
    self._target(*self._args, **self._kwargs)
  File &quot;distributed/test_distributed.py&quot;, line 2126, in _run
    getattr(self, self.id().split(&quot;.&quot;)[2])()
  File &quot;distributed/test_distributed.py&quot;, line 2052, in wrapper
    fn(self)
  File &quot;distributed/test_distributed.py&quot;, line 119, in wrapper
    return func(*args, **kwargs)
  File &quot;distributed/test_distributed.py&quot;, line 135, in wrapper
    return func(*args, **kwargs)
  File &quot;/var/lib/jenkins/.local/lib/python3.6/site-packages/torch/testing/_internal/common_distributed.py&quot;, line 102, in wrapper
    return func(*args, **kwargs)
  File &quot;distributed/test_distributed.py&quot;, line 1878, in test_DistributedDataParallel
    self._test_DistributedDataParallel(gpu_subset=gpus, rank=rank)
  File &quot;distributed/test_distributed.py&quot;, line 1812, in _test_DistributedDataParallel
    model_DDP, device_ids=gpu_subset
  File &quot;/var/lib/jenkins/.local/lib/python3.6/site-packages/torch/nn/parallel/distributed.py&quot;, line 305, in __init__
    self._ddp_init_helper()
  File &quot;/var/lib/jenkins/.local/lib/python3.6/site-packages/torch/nn/parallel/distributed.py&quot;, line 387, in _ddp_init_helper
    expect_sparse_gradient)
RuntimeError: Model replicas must have an equal number of parameters."><pre class="notranslate"><code class="notranslate">Process process 0:
Traceback (most recent call last):
  File "/opt/conda/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/opt/conda/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "distributed/test_distributed.py", line 2126, in _run
    getattr(self, self.id().split(".")[2])()
  File "distributed/test_distributed.py", line 2052, in wrapper
    fn(self)
  File "distributed/test_distributed.py", line 119, in wrapper
    return func(*args, **kwargs)
  File "distributed/test_distributed.py", line 135, in wrapper
    return func(*args, **kwargs)
  File "/var/lib/jenkins/.local/lib/python3.6/site-packages/torch/testing/_internal/common_distributed.py", line 102, in wrapper
    return func(*args, **kwargs)
  File "distributed/test_distributed.py", line 1878, in test_DistributedDataParallel
    self._test_DistributedDataParallel(gpu_subset=gpus, rank=rank)
  File "distributed/test_distributed.py", line 1812, in _test_DistributedDataParallel
    model_DDP, device_ids=gpu_subset
  File "/var/lib/jenkins/.local/lib/python3.6/site-packages/torch/nn/parallel/distributed.py", line 305, in __init__
    self._ddp_init_helper()
  File "/var/lib/jenkins/.local/lib/python3.6/site-packages/torch/nn/parallel/distributed.py", line 387, in _ddp_init_helper
    expect_sparse_gradient)
RuntimeError: Model replicas must have an equal number of parameters.
</code></pre></div>
<p dir="auto">If we limit the number of devices as follows:<br>
<code class="notranslate">HIP_VISIBLE_DEVICES=0,1 BACKEND=nccl WORLD_SIZE=2 TEMP_DIR=/tmp python3.6 test_distributed.py --verbose TestDistBackend.test_DistributedDataParallel</code><br>
the test passes.</p>
<h2 dir="auto">Expected behavior</h2>
<p dir="auto">Test should pass.</p>
<h2 dir="auto">Environment</h2>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Collecting environment information...
PyTorch version: 1.5.0a0+f5e5569
Is debug build: No
CUDA used to build PyTorch: Could not collect

OS: Ubuntu 16.04.6 LTS
GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.12) 5.4.0 20160609
CMake version: version 3.6.3

Python version: 3.6
Is CUDA available: Yes
CUDA runtime version: Could not collect
GPU models and configuration: Could not collect
Nvidia driver version: Could not collect
cuDNN version: Could not collect
ROCm version: 3.1

Versions of relevant libraries:
[pip] numpy==1.18.1
[pip] torch==1.5.0a0+f5e5569
[pip] torchvision==0.5.0a0+44a5bae
[conda] blas                      1.0                         mkl
[conda] mkl                       2020.0                      166
[conda] mkl-include               2020.0                      166
[conda] mkl-service               2.3.0            py36he904b0f_0
[conda] mkl_fft                   1.0.15           py36ha843d7b_0
[conda] mkl_random                1.1.0            py36hd6b4f25_0"><pre class="notranslate"><code class="notranslate">Collecting environment information...
PyTorch version: 1.5.0a0+f5e5569
Is debug build: No
CUDA used to build PyTorch: Could not collect

OS: Ubuntu 16.04.6 LTS
GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.12) 5.4.0 20160609
CMake version: version 3.6.3

Python version: 3.6
Is CUDA available: Yes
CUDA runtime version: Could not collect
GPU models and configuration: Could not collect
Nvidia driver version: Could not collect
cuDNN version: Could not collect
ROCm version: 3.1

Versions of relevant libraries:
[pip] numpy==1.18.1
[pip] torch==1.5.0a0+f5e5569
[pip] torchvision==0.5.0a0+44a5bae
[conda] blas                      1.0                         mkl
[conda] mkl                       2020.0                      166
[conda] mkl-include               2020.0                      166
[conda] mkl-service               2.3.0            py36he904b0f_0
[conda] mkl_fft                   1.0.15           py36ha843d7b_0
[conda] mkl_random                1.1.0            py36hd6b4f25_0
</code></pre></div>
<p dir="auto">cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ezyang/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ezyang">@ezyang</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/gchanan/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/gchanan">@gchanan</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/zou3519/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/zou3519">@zou3519</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/pietern/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/pietern">@pietern</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/mrshenli/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/mrshenli">@mrshenli</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/pritamdamania87/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/pritamdamania87">@pritamdamania87</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/zhaojuanmao/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/zhaojuanmao">@zhaojuanmao</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/satgera/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/satgera">@satgera</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/rohan-varma/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/rohan-varma">@rohan-varma</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/gqchen/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/gqchen">@gqchen</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/aazzolini/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/aazzolini">@aazzolini</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/xush6528/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/xush6528">@xush6528</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/osalpekar/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/osalpekar">@osalpekar</a></p>