<p dir="auto">I'm using python shared memory from multiprocessing package to share numpy arrays between processes.<br>
There is a caveat, however, that populating such objects from buffer requires duplicate copy operation.<br>
np.frombuffer creates new array and there is no option to populate existing array which is linked with a shared memory buffer.<br>
In result, it is required to run 2 operations: deserializing the buffer content to numpy object and the second to copy elements from that array to the one linked with shared memory.<br>
I would like to propose extending np.frombuffer to take additional argument of existing numpy array that would be populated with the content instead of creating new one.</p>
<h3 dir="auto">Reproducing code example:</h3>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import numpy as np
from multiprocessing import shared_memory

input = np.ones((1,10,10,10))
b = input.tobytes()

input_shm = shared_memory.SharedMemory(create=True, size=input.nbytes)
write_array = np.ndarray(input.shape, dtype=input.dtype,buffer=input_shm.buf)

content = np.frombuffer(b,dtype=input.dtype).reshape(input.shape) # this is one copy extensive operation

write_array[:] = content[:]  # this is duplicate copy to populate shared memory buffer

restored_numpy = np.ndarray(input.shape, dtype=input.dtype,buffer=input_shm.buf)
# reading from shared memory is quick - attaching the memory block

print(&quot;restored&quot;,restored_numpy)
"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">numpy</span> <span class="pl-k">as</span> <span class="pl-s1">np</span>
<span class="pl-k">from</span> <span class="pl-s1">multiprocessing</span> <span class="pl-k">import</span> <span class="pl-s1">shared_memory</span>

<span class="pl-s1">input</span> <span class="pl-c1">=</span> <span class="pl-s1">np</span>.<span class="pl-en">ones</span>((<span class="pl-c1">1</span>,<span class="pl-c1">10</span>,<span class="pl-c1">10</span>,<span class="pl-c1">10</span>))
<span class="pl-s1">b</span> <span class="pl-c1">=</span> <span class="pl-s1">input</span>.<span class="pl-en">tobytes</span>()

<span class="pl-s1">input_shm</span> <span class="pl-c1">=</span> <span class="pl-s1">shared_memory</span>.<span class="pl-v">SharedMemory</span>(<span class="pl-s1">create</span><span class="pl-c1">=</span><span class="pl-c1">True</span>, <span class="pl-s1">size</span><span class="pl-c1">=</span><span class="pl-s1">input</span>.<span class="pl-s1">nbytes</span>)
<span class="pl-s1">write_array</span> <span class="pl-c1">=</span> <span class="pl-s1">np</span>.<span class="pl-en">ndarray</span>(<span class="pl-s1">input</span>.<span class="pl-s1">shape</span>, <span class="pl-s1">dtype</span><span class="pl-c1">=</span><span class="pl-s1">input</span>.<span class="pl-s1">dtype</span>,<span class="pl-s1">buffer</span><span class="pl-c1">=</span><span class="pl-s1">input_shm</span>.<span class="pl-s1">buf</span>)

<span class="pl-s1">content</span> <span class="pl-c1">=</span> <span class="pl-s1">np</span>.<span class="pl-en">frombuffer</span>(<span class="pl-s1">b</span>,<span class="pl-s1">dtype</span><span class="pl-c1">=</span><span class="pl-s1">input</span>.<span class="pl-s1">dtype</span>).<span class="pl-en">reshape</span>(<span class="pl-s1">input</span>.<span class="pl-s1">shape</span>) <span class="pl-c"># this is one copy extensive operation</span>

<span class="pl-s1">write_array</span>[:] <span class="pl-c1">=</span> <span class="pl-s1">content</span>[:]  <span class="pl-c"># this is duplicate copy to populate shared memory buffer</span>

<span class="pl-s1">restored_numpy</span> <span class="pl-c1">=</span> <span class="pl-s1">np</span>.<span class="pl-en">ndarray</span>(<span class="pl-s1">input</span>.<span class="pl-s1">shape</span>, <span class="pl-s1">dtype</span><span class="pl-c1">=</span><span class="pl-s1">input</span>.<span class="pl-s1">dtype</span>,<span class="pl-s1">buffer</span><span class="pl-c1">=</span><span class="pl-s1">input_shm</span>.<span class="pl-s1">buf</span>)
<span class="pl-c"># reading from shared memory is quick - attaching the memory block</span>

<span class="pl-en">print</span>(<span class="pl-s">"restored"</span>,<span class="pl-s1">restored_numpy</span>)</pre></div>
<p dir="auto">The desired behaviour would be like:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="np.frombuffer(b,dtype=input.dtype,target_ndarray=write_array) 
# deserialization and populating the shared memory buffer at one time
"><pre class="notranslate"><span class="pl-s1">np</span>.<span class="pl-en">frombuffer</span>(<span class="pl-s1">b</span>,<span class="pl-s1">dtype</span><span class="pl-c1">=</span><span class="pl-s1">input</span>.<span class="pl-s1">dtype</span>,<span class="pl-s1">target_ndarray</span><span class="pl-c1">=</span><span class="pl-s1">write_array</span>) 
<span class="pl-c"># deserialization and populating the shared memory buffer at one time</span></pre></div>
<h3 dir="auto">Numpy/Python version information:</h3>

<p dir="auto">import sys, numpy; print(numpy.<strong>version</strong>, sys.version)<br>
1.18.1 3.8.1 (default, Jan 22 2020, 16:31:49)<br>
[GCC 9.2.1 20200111 gcc-9-branch@280154]</p>
<p dir="auto">I ran into this, and the solution found on <a href="http://mail.scipy.org/pipermail/numpy-discussion/2012-November/064485.html" rel="nofollow">numpy-discussion</a> worked for me, but does<br>
turn up on a search of the issues.</p>
<p dir="auto">Summary:</p>
<ul dir="auto">
<li>Python &lt;3.2 are compiled with VS2008 (SDK v7.0,"sdk for win7 + NET 3.5.1")</li>
<li>Python 3.3 (+?) is compiled with VS2010 (SDK v7.1, "sdk for win7 + NET 4")</li>
<li>It's often recommended that c extensions (such as numpy) be compiled using the same compiler<br>
used to compile the python binaries, to ensure CRT compatability, so using VS2010/SDK 7.1 to compile<br>
numpy for python 3.3 is implied.</li>
<li>VS2010 (msvc10) behaves differently with regards to manifest files then VS2008 (msvc9) but the cpython distutils, which the numpy build process relies on,<br>
have not been altered to allow for this (as of python 3.3.3).</li>
<li>a <a href="http://bugs.python.org/issue16296" rel="nofollow">cpython issue</a> was filed in dec/2012, but has stalled.</li>
<li>The issue manifests as a "broken toolchain" exception when attempting to build<br>
numpy (both 1.7.1 and 1.8.0), which results from the following compiler error message.</li>
</ul>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Found executable C:\Program Files\Microsoft SDKs\Windows\v7.0A\bin\mt.exe

_configtest.exe.manifest : general error c1010070: Failed to load and
parse the manifest. The system cannot find the file specified."><pre class="notranslate"><code class="notranslate">Found executable C:\Program Files\Microsoft SDKs\Windows\v7.0A\bin\mt.exe

_configtest.exe.manifest : general error c1010070: Failed to load and
parse the manifest. The system cannot find the file specified.
</code></pre></div>
<p dir="auto">The fix suggested by Christoph Gohlke on the ml is to edit the file <a href="https://github.com/akheron/cpython/blob/157e2d3078abe02576dd8710a8a4104dd620ccbe/Lib/distutils/msvc9compiler.py#L648">"Lib/distutils/msvc9compiler.py:648"</a>, Located under your python 3.3 install directory, and modify the line from:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="mfinfo = self.manifest_get_embed_info(target_desc, ld_args)"><pre class="notranslate"><code class="notranslate">mfinfo = self.manifest_get_embed_info(target_desc, ld_args)
</code></pre></div>
<p dir="auto">to</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="mfinfo = None"><pre class="notranslate"><code class="notranslate">mfinfo = None
</code></pre></div>
<p dir="auto">I've tried, and it has worked for me. Thanks goes to <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/cgohlke/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/cgohlke">@cgohlke</a>, the intrepid crusader for windows pydata binaries.</p>
<p dir="auto">There, that should give pagerank something to chew on.</p>