<p dir="auto">Log:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="21:31:08 test_proper_exit (__main__.TestDataLoader)
21:31:08 There might be ConnectionResetError or leaked semaphore warning (due to dirty process exit), but they are all safe to ignore ... Traceback (most recent call last):
21:31:10   File &quot;&lt;string&gt;&quot;, line 1, in &lt;module&gt;
21:31:10   File &quot;C:\Jenkins\Miniconda3\lib\multiprocessing\spawn.py&quot;, line 105, in spawn_main
21:31:10     exitcode = _main(fd)
21:31:10   File &quot;C:\Jenkins\Miniconda3\lib\multiprocessing\spawn.py&quot;, line 115, in _main
21:31:10     self = reduction.pickle.load(from_parent)
21:31:10   File &quot;C:\Jenkins\Miniconda3\lib\multiprocessing\connection.py&quot;, line 942, in rebuild_pipe_connection
21:31:10     handle = dh.detach()
21:31:10   File &quot;C:\Jenkins\Miniconda3\lib\multiprocessing\reduction.py&quot;, line 129, in detach
21:31:10     self._access, False, _winapi.DUPLICATE_CLOSE_SOURCE)
21:31:10 PermissionError: [WinError 5] Access is denied
21:31:25 FAIL

21:32:15 ======================================================================
21:32:15 FAIL: test_proper_exit (__main__.TestDataLoader)
21:32:15 There might be ConnectionResetError or leaked semaphore warning (due to dirty process exit), but they are all safe to ignore
21:32:15 ----------------------------------------------------------------------
21:32:15 Traceback (most recent call last):
21:32:15   File &quot;C:\Jenkins\workspace\pytorch-builds\pytorch-win-ws2016-cuda9-cudnn7-py3-test2\test\common_utils.py&quot;, line 117, in wrapper
21:32:15     fn(*args, **kwargs)
21:32:15   File &quot;test_dataloader.py&quot;, line 726, in test_proper_exit
21:32:15     self.assertTrue(setup_event.is_set(), 'loader process setup timed out')
21:32:15 AssertionError: False is not true : loader process setup timed out
21:32:15 
21:32:15 ----------------------------------------------------------------------"><pre class="notranslate"><code class="notranslate">21:31:08 test_proper_exit (__main__.TestDataLoader)
21:31:08 There might be ConnectionResetError or leaked semaphore warning (due to dirty process exit), but they are all safe to ignore ... Traceback (most recent call last):
21:31:10   File "&lt;string&gt;", line 1, in &lt;module&gt;
21:31:10   File "C:\Jenkins\Miniconda3\lib\multiprocessing\spawn.py", line 105, in spawn_main
21:31:10     exitcode = _main(fd)
21:31:10   File "C:\Jenkins\Miniconda3\lib\multiprocessing\spawn.py", line 115, in _main
21:31:10     self = reduction.pickle.load(from_parent)
21:31:10   File "C:\Jenkins\Miniconda3\lib\multiprocessing\connection.py", line 942, in rebuild_pipe_connection
21:31:10     handle = dh.detach()
21:31:10   File "C:\Jenkins\Miniconda3\lib\multiprocessing\reduction.py", line 129, in detach
21:31:10     self._access, False, _winapi.DUPLICATE_CLOSE_SOURCE)
21:31:10 PermissionError: [WinError 5] Access is denied
21:31:25 FAIL

21:32:15 ======================================================================
21:32:15 FAIL: test_proper_exit (__main__.TestDataLoader)
21:32:15 There might be ConnectionResetError or leaked semaphore warning (due to dirty process exit), but they are all safe to ignore
21:32:15 ----------------------------------------------------------------------
21:32:15 Traceback (most recent call last):
21:32:15   File "C:\Jenkins\workspace\pytorch-builds\pytorch-win-ws2016-cuda9-cudnn7-py3-test2\test\common_utils.py", line 117, in wrapper
21:32:15     fn(*args, **kwargs)
21:32:15   File "test_dataloader.py", line 726, in test_proper_exit
21:32:15     self.assertTrue(setup_event.is_set(), 'loader process setup timed out')
21:32:15 AssertionError: False is not true : loader process setup timed out
21:32:15 
21:32:15 ----------------------------------------------------------------------
</code></pre></div>
<p dir="auto">Context: <a href="https://ci.pytorch.org/jenkins/job/pytorch-builds/job/pytorch-win-ws2016-cuda9-cudnn7-py3-test2/16532/console" rel="nofollow">https://ci.pytorch.org/jenkins/job/pytorch-builds/job/pytorch-win-ws2016-cuda9-cudnn7-py3-test2/16532/console</a> <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="385426715" data-permission-text="Title is private" data-url="https://github.com/pytorch/pytorch/issues/14490" data-hovercard-type="pull_request" data-hovercard-url="/pytorch/pytorch/pull/14490/hovercard" href="https://github.com/pytorch/pytorch/pull/14490">#14490</a></p>
<h2 dir="auto"><g-emoji class="g-emoji" alias="rocket" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f680.png">🚀</g-emoji> Feature</h2>

<p dir="auto">When I try to load a plugin library built with 2 cpp file,</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="torch.ops.load_library(&quot;libtorch_plugins.so&quot;)"><pre class="notranslate"><span class="pl-s1">torch</span>.<span class="pl-s1">ops</span>.<span class="pl-en">load_library</span>(<span class="pl-s">"libtorch_plugins.so"</span>)</pre></div>
<p dir="auto">I got the following error:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Only a single TORCH_LIBRARY can be used to register the namespace decode; please put all of your definitions in a single TORCH_LIBRARY block.  If you were trying to specify implementations, consider using TORCH_LIBRARY_IMPL (which can be duplicated).  Previous registration of TORCH_LIBRARY was registered at /home/cloudhan/workspaces/torch_plugins/src/readjpeg_cpu.cpp:146; latest registration was registered at /home/cloudhan/workspaces/torch_plugins/src/readpng_cpu.cpp:87"><pre class="notranslate"><code class="notranslate">Only a single TORCH_LIBRARY can be used to register the namespace decode; please put all of your definitions in a single TORCH_LIBRARY block.  If you were trying to specify implementations, consider using TORCH_LIBRARY_IMPL (which can be duplicated).  Previous registration of TORCH_LIBRARY was registered at /home/cloudhan/workspaces/torch_plugins/src/readjpeg_cpu.cpp:146; latest registration was registered at /home/cloudhan/workspaces/torch_plugins/src/readpng_cpu.cpp:87
</code></pre></div>
<h2 dir="auto">Motivation</h2>

<p dir="auto">Currently, the only valid way to build a plugin library is to collect all op registration into one translation unit.<br>
That is</p>
<div class="highlight highlight-source-c++ notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="// file foo.cpp

// define op foo"><pre class="notranslate"><span class="pl-c"><span class="pl-c">//</span> file foo.cpp</span>

<span class="pl-c"><span class="pl-c">//</span> define op foo</span></pre></div>
<div class="highlight highlight-source-c++ notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="// file bar.cpp

// define op bar"><pre class="notranslate"><span class="pl-c"><span class="pl-c">//</span> file bar.cpp</span>

<span class="pl-c"><span class="pl-c">//</span> define op bar</span></pre></div>
<div class="highlight highlight-source-c++ notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="// registration.cpp

#include &quot;foo.h&quot;
#include &quot;bar.h&quot;

// register ops foo and bar "><pre class="notranslate"><span class="pl-c"><span class="pl-c">//</span> registration.cpp</span>

#<span class="pl-k">include</span> <span class="pl-s"><span class="pl-pds">"</span>foo.h<span class="pl-pds">"</span></span>
#<span class="pl-k">include</span> <span class="pl-s"><span class="pl-pds">"</span>bar.h<span class="pl-pds">"</span></span>

<span class="pl-c"><span class="pl-c">//</span> register ops foo and bar </span></pre></div>
<p dir="auto">In this way, it is hard to maintain flexible extension op set. Since to add a new op,  you need to directly modify <code class="notranslate">registration.cpp</code><br>
To remove unneeded ops, you also need to modify <code class="notranslate">registration.cpp</code></p>
<p dir="auto">If distributed registration is supported, all I need is</p>
<div class="highlight highlight-source-c++ notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="// file foo.cpp

// define op foo
// register op foo"><pre class="notranslate"><span class="pl-c"><span class="pl-c">//</span> file foo.cpp</span>

<span class="pl-c"><span class="pl-c">//</span> define op foo</span>
<span class="pl-c"><span class="pl-c">//</span> register op foo</span></pre></div>
<div class="highlight highlight-source-c++ notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="// file bar.cpp

// define op bar
// register op bar "><pre class="notranslate"><span class="pl-c"><span class="pl-c">//</span> file bar.cpp</span>

<span class="pl-c"><span class="pl-c">//</span> define op bar</span>
<span class="pl-c"><span class="pl-c">//</span> register op bar </span></pre></div>
<p dir="auto">Simply control which cpp file to be compiled and linked, I can now control the registration easier.</p>
<p dir="auto">cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/yf225/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/yf225">@yf225</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/glaringlee/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/glaringlee">@glaringlee</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ezyang/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ezyang">@ezyang</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/bhosmer/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/bhosmer">@bhosmer</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/smessmer/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/smessmer">@smessmer</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ljk53/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ljk53">@ljk53</a></p>