<h2 dir="auto"><g-emoji class="g-emoji" alias="rocket" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f680.png">🚀</g-emoji> Feature/Enhnacement</h2>
<p dir="auto">During RPC initialization, each participating process initializes a singleton <code class="notranslate">DistAutogradContainer</code>. During shutdown, we cleanup items such as the RRef context and RPC agent, but do not reset this singleton dist autograd container. This prevents RPC from being reinitialized (i.e. a group of processes cannot call <code class="notranslate">rpc.shutdown() ; dist.barrier() ; rpc.init_rpc(...)</code>. It will fail with the following error:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="/test/distributed/rpc/faulty_agent/rpc_spawn_faulty#binary,link-tree
/torch/distributed/rpc/__init__.py&quot;, line 85, in init_rpc
&gt;     dist_autograd._init(rank)
&gt; RuntimeError: Container is already initialized! Cannot initialize it twice!"><pre class="notranslate"><code class="notranslate">/test/distributed/rpc/faulty_agent/rpc_spawn_faulty#binary,link-tree
/torch/distributed/rpc/__init__.py", line 85, in init_rpc
&gt;     dist_autograd._init(rank)
&gt; RuntimeError: Container is already initialized! Cannot initialize it twice!
</code></pre></div>
<h2 dir="auto">Motivation</h2>
<p dir="auto">I was looking into seeing if we could reuse spawned subprocesses across our RPC tests, which would decrease the amount of time it takes to run the RPC test suite; however, this would require processes to be able to re-init RPC, which is blocked by this error. Also, I was writing a set of similar tests in <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="619349996" data-permission-text="Title is private" data-url="https://github.com/pytorch/pytorch/issues/38590" data-hovercard-type="pull_request" data-hovercard-url="/pytorch/pytorch/pull/38590/hovercard" href="https://github.com/pytorch/pytorch/pull/38590">#38590</a>, and wanted to use this pattern to reduce code duplication, but couldn't (the tests need to use different initializations of RPC since they test the RRef deletion behavior triggered by the shutdown sequence).</p>
<h2 dir="auto">Pitch</h2>
<p dir="auto">I don't know if we necessarily need to destroy the container, maybe we could just de-initialize it, and then re-init if RPC is being started back up again.</p>
<p dir="auto">cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/pietern/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/pietern">@pietern</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/mrshenli/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/mrshenli">@mrshenli</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/pritamdamania87/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/pritamdamania87">@pritamdamania87</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/zhaojuanmao/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/zhaojuanmao">@zhaojuanmao</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/satgera/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/satgera">@satgera</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/gqchen/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/gqchen">@gqchen</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/aazzolini/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/aazzolini">@aazzolini</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/rohan-varma/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/rohan-varma">@rohan-varma</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/xush6528/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/xush6528">@xush6528</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/jjlilley/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/jjlilley">@jjlilley</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/osalpekar/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/osalpekar">@osalpekar</a></p>
<ul dir="auto">
<li>
<p dir="auto">OS: Ubuntu 16.04</p>
</li>
<li>
<p dir="auto">PyTorch version: source code@master</p>
</li>
<li>
<p dir="auto">How you installed PyTorch (conda, pip, source): source</p>
</li>
<li>
<p dir="auto">Python version: Python3.6@conda</p>
</li>
<li>
<p dir="auto">CUDA/cuDNN version: 9.0</p>
</li>
<li>
<p dir="auto">GPU models and configuration: 4x1080Ti</p>
</li>
<li>
<p dir="auto">GCC version (if compiling from source): gcc5.4</p>
</li>
<li>
<p dir="auto">A script to reproduce the bug. Please try to provide as minimal of a test case as possible.<br>
dataloader(..., numworkers=2)<br>
then iterate over dataset</p>
</li>
<li>
<p dir="auto">Error messages and/or stack traces of the bug<br>
Files already downloaded and verified<br>
Files already downloaded and verified<br>
ERROR: Unexpected segmentation fault encountered in worker.<br>
ERROR: Unexpected segmentation fault encountered in worker.<br>
Traceback (most recent call last):<br>
File "./main.py", line 445, in <br>
main()<br>
File "./main.py", line 426, in main<br>
train_TR(dataloader, dataloader_test, net, loss_f, opt.lr, opt.radius, opt.max_inner, opt.epoch)<br>
File "./main.py", line 214, in train_TR<br>
for x, y in data_iter:<br>
File "/home/luinx/.local/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 275, in <strong>next</strong><br>
idx, batch = self._get_batch()<br>
File "/home/luinx/.local/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 254, in _get_batch<br>
return self.data_queue.get()<br>
File "/home/luinx/miniconda3/lib/python3.6/multiprocessing/queues.py", line 335, in get<br>
res = self._reader.recv_bytes()<br>
File "/home/luinx/miniconda3/lib/python3.6/multiprocessing/connection.py", line 216, in recv_bytes<br>
buf = self._recv_bytes(maxlength)<br>
File "/home/luinx/miniconda3/lib/python3.6/multiprocessing/connection.py", line 407, in _recv_bytes<br>
buf = self._recv(4)<br>
File "/home/luinx/miniconda3/lib/python3.6/multiprocessing/connection.py", line 379, in _recv<br>
chunk = read(handle, remaining)<br>
File "/home/luinx/.local/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 175, in handler<br>
_error_if_any_worker_fails()<br>
RuntimeError: DataLoader worker (pid 21010) is killed by signal: Segmentation fault.</p>
</li>
<li>
<p dir="auto">Context around what you are trying to do<br>
Nothing special, just loading data. When I set numworkers=0, then it works fine. I guess it is triggered by interprocess communication.</p>
</li>
</ul>