<p dir="auto">I have ran into this error on the parameter broadcast step of <code class="notranslate">torch.nn.parallel.replicate</code>.<br>
It sounds like this, to be precise: <code class="notranslate">NCCL Error 1: unhandled cuda error</code>.<br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="248647185" data-permission-text="Title is private" data-url="https://github.com/pytorch/pytorch/issues/2332" data-hovercard-type="issue" data-hovercard-url="/pytorch/pytorch/issues/2332/hovercard" href="https://github.com/pytorch/pytorch/issues/2332">#2332</a> is possibly related.</p>
<h2 dir="auto"><g-emoji class="g-emoji" alias="bug" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f41b.png">🐛</g-emoji> Bug</h2>
<p dir="auto">A module might declare a buffer of unknown size in its init function by using:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="self.register_buffer('name', None)"><pre class="notranslate"><span class="pl-s1">self</span>.<span class="pl-en">register_buffer</span>(<span class="pl-s">'name'</span>, <span class="pl-c1">None</span>)</pre></div>
<p dir="auto">The buffer might later contain a valid tensor. Saving this (now non-None) buffer with <code class="notranslate">state_dict()</code> works but loading this state dict to a newly instantiated module (where the buffer is currently None) with <code class="notranslate">load_state_dict()</code> fails as the buffer is unexpected.</p>
<h2 dir="auto">To Reproduce</h2>
<p dir="auto">Steps to reproduce the behavior:</p>
<ol dir="auto">
<li>Define a module with an empty buffer in the init function.</li>
<li>Set the buffer to something valid (e.g., in <code class="notranslate">forward</code>).</li>
<li>Call <code class="notranslate">state_dict()</code></li>
<li>Instantiated a second instance of the module</li>
<li>call <code class="notranslate">load_state_dict()</code> on the second module</li>
</ol>

<h2 dir="auto">Expected behavior</h2>

<p dir="auto"><code class="notranslate">load_state_dict()</code> replaces the currently None buffer with the non-None saved buffer from the state_dict.</p>
<h2 dir="auto">Environment</h2>
<p dir="auto">Collecting environment information...<br>
PyTorch version: 1.2.0<br>
Is debug build: No<br>
CUDA used to build PyTorch: 10.0.130</p>
<p dir="auto">OS: CentOS Linux 7 (Core)<br>
GCC version: (GCC) 4.8.5 20150623 (Red Hat 4.8.5-16)<br>
CMake version: version 2.8.12.2</p>
<p dir="auto">Python version: 3.7<br>
Is CUDA available: No<br>
CUDA runtime version: No CUDA<br>
GPU models and configuration: No CUDA<br>
Nvidia driver version: No CUDA<br>
cuDNN version: No CUDA</p>
<p dir="auto">Versions of relevant libraries:<br>
[pip] msgpack-numpy==0.4.4.3<br>
[pip] numpy==1.17.2<br>
[pip] pytorch-pretrained-bert==0.6.2<br>
[pip] torch==1.2.0<br>
[pip] torchvision==0.4.0a0<br>
[conda] _pytorch_select           0.2                       gpu_0<br>
[conda] blas                      1.0                         mkl<br>
[conda] mkl                       2019.4                      243<br>
[conda] mkl-service               2.3.0            py37he904b0f_0<br>
[conda] mkl_fft                   1.0.14           py37ha843d7b_0<br>
[conda] mkl_random                1.1.0            py37hd6b4f25_0<br>
[conda] pytorch                   1.2.0           py3.7_cuda10.0.130_cudnn7.6.2_0    pytorch<br>
[conda] pytorch-pretrained-bert   0.6.2                    pypi_0    pypi<br>
[conda] torch                     1.0.1.post2              pypi_0    pypi<br>
[conda] torchvision               0.2.2.post3              pypi_0    pypi</p>