<p dir="auto"><em>Original ticket <a href="http://projects.scipy.org/numpy/ticket/1036" rel="nofollow">http://projects.scipy.org/numpy/ticket/1036</a> on 2009-03-04 by <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/rc/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/rc">@rc</a>, assigned to <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/rc/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/rc">@rc</a>.</em></p>
<p dir="auto">Neil Crighton wrote:</p>
<p dir="auto">As far as I know there isn't an obvious way to get the functionality of setmember1d working on non-unique inputs. However, I've needed this operation quite a lot, so here's a function I wrote that does it. It's only a few times slower than numpy's setmember1d. You're welcome to use it.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="import numpy as np

def ismember(a1,a2):
    &quot;&quot;&quot; Test whether items from a2 are in a1.

    This does the same thing as np.setmember1d, but works on
    non-unique arrays.

    Only a few (2-4) times slower than np.setmember1d, and a lot
    faster than [i in a2 for i in a1].

    An example that np.setmember1d gets wrong: 

    &gt;&gt;&gt; a1 = np.array([5,4,5,3,4,4,3,4,3,5,2,1,5,5])
    &gt;&gt;&gt; a2 = [2,3,4]
    &gt;&gt;&gt; mask = ismember(a1,a2)
    &gt;&gt;&gt; a1[mask]
    array([4, 3, 4, 4, 3, 4, 3, 2])
    &quot;&quot;&quot;
    a2 = set(a2)
    a1 = np.asarray(a1)
    ind = a1.argsort()
    a1 = a1[ind]
    mask  = []
    # need this bit because prev is not defined for first item
    item  = a1[0]
    if item in a2:
        mask.append(True)
        a2.remove(item)
    else:
        mask.append(False)
    prev = item
    # main loop
    for item in a1[1:]:
        if item == prev:
            mask.append(mask[-1])
        elif item in a2:
            mask.append(True)
            prev = item
            a2.remove(item)
        else:
            mask.append(False)
            prev = item
    # restore mask to original ordering of a1 and return
    mask = np.array(mask)
    return mask[ind.argsort()]"><pre class="notranslate"><code class="notranslate">import numpy as np

def ismember(a1,a2):
    """ Test whether items from a2 are in a1.

    This does the same thing as np.setmember1d, but works on
    non-unique arrays.

    Only a few (2-4) times slower than np.setmember1d, and a lot
    faster than [i in a2 for i in a1].

    An example that np.setmember1d gets wrong: 

    &gt;&gt;&gt; a1 = np.array([5,4,5,3,4,4,3,4,3,5,2,1,5,5])
    &gt;&gt;&gt; a2 = [2,3,4]
    &gt;&gt;&gt; mask = ismember(a1,a2)
    &gt;&gt;&gt; a1[mask]
    array([4, 3, 4, 4, 3, 4, 3, 2])
    """
    a2 = set(a2)
    a1 = np.asarray(a1)
    ind = a1.argsort()
    a1 = a1[ind]
    mask  = []
    # need this bit because prev is not defined for first item
    item  = a1[0]
    if item in a2:
        mask.append(True)
        a2.remove(item)
    else:
        mask.append(False)
    prev = item
    # main loop
    for item in a1[1:]:
        if item == prev:
            mask.append(mask[-1])
        elif item in a2:
            mask.append(True)
            prev = item
            a2.remove(item)
        else:
            mask.append(False)
            prev = item
    # restore mask to original ordering of a1 and return
    mask = np.array(mask)
    return mask[ind.argsort()]
</code></pre></div>
<h3 dir="auto">Describe the issue:</h3>
<p dir="auto">A simple code snippet below calculating bins for a given input fails while trying to allocate 12.7 PiB of RAM. Probably related to my floats precision, but is definitely not the expected result.</p>
<p dir="auto">I would be most grateful if anyone could suggest a workaround while this bug is being fixed.</p>
<h3 dir="auto">Reproduce the code example:</h3>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import numpy as np
x = np.array([6.3,
 6.3,
 7.399999999999999,
 7.399999999999999,
 7.399999999999999,
 7.399999999999999,
 7.4,
 7.4,
 7.4,
 7.4,
 7.4,
 7.4,
 7.4,
 7.4,
 7.4,
 7.4,
 7.4,
 7.4,
 7.4,
 7.407142857142857,
 7.407142857142857,
 7.4125,
 7.4125])
np.histogram(x, density=True, bins=&quot;auto&quot;)"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">numpy</span> <span class="pl-k">as</span> <span class="pl-s1">np</span>
<span class="pl-s1">x</span> <span class="pl-c1">=</span> <span class="pl-s1">np</span>.<span class="pl-en">array</span>([<span class="pl-c1">6.3</span>,
 <span class="pl-c1">6.3</span>,
 <span class="pl-c1">7.399999999999999</span>,
 <span class="pl-c1">7.399999999999999</span>,
 <span class="pl-c1">7.399999999999999</span>,
 <span class="pl-c1">7.399999999999999</span>,
 <span class="pl-c1">7.4</span>,
 <span class="pl-c1">7.4</span>,
 <span class="pl-c1">7.4</span>,
 <span class="pl-c1">7.4</span>,
 <span class="pl-c1">7.4</span>,
 <span class="pl-c1">7.4</span>,
 <span class="pl-c1">7.4</span>,
 <span class="pl-c1">7.4</span>,
 <span class="pl-c1">7.4</span>,
 <span class="pl-c1">7.4</span>,
 <span class="pl-c1">7.4</span>,
 <span class="pl-c1">7.4</span>,
 <span class="pl-c1">7.4</span>,
 <span class="pl-c1">7.407142857142857</span>,
 <span class="pl-c1">7.407142857142857</span>,
 <span class="pl-c1">7.4125</span>,
 <span class="pl-c1">7.4125</span>])
<span class="pl-s1">np</span>.<span class="pl-en">histogram</span>(<span class="pl-s1">x</span>, <span class="pl-s1">density</span><span class="pl-c1">=</span><span class="pl-c1">True</span>, <span class="pl-s1">bins</span><span class="pl-c1">=</span><span class="pl-s">"auto"</span>)</pre></div>
<h3 dir="auto">Error message:</h3>
<div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="Traceback (most recent call last):
  File &quot;/Users/user/miniforge3/envs/clients/lib/python3.8/site-packages/IPython/core/interactiveshell.py&quot;, line 3433, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File &quot;&lt;ipython-input-3-f76ea541d063&gt;&quot;, line 25, in &lt;module&gt;
    np.histogram(x, density=True, bins=&quot;auto&quot;)
  File &quot;&lt;__array_function__ internals&gt;&quot;, line 180, in histogram
  File &quot;/Users/user/miniforge3/envs/clients/lib/python3.8/site-packages/numpy/lib/histograms.py&quot;, line 793, in histogram
    bin_edges, uniform_bins = _get_bin_edges(a, bins, range, weights)
  File &quot;/Users/user/miniforge3/envs/clients/lib/python3.8/site-packages/numpy/lib/histograms.py&quot;, line 446, in _get_bin_edges
    bin_edges = np.linspace(
  File &quot;&lt;__array_function__ internals&gt;&quot;, line 180, in linspace
  File &quot;/Users/user/miniforge3/envs/clients/lib/python3.8/site-packages/numpy/core/function_base.py&quot;, line 135, in linspace
    y = _nx.arange(0, num, dtype=dt).reshape((-1,) + (1,) * ndim(delta))
numpy.core._exceptions._ArrayMemoryError: Unable to allocate 12.7 PiB for an array with shape (1781062197026280,) and data type float64"><pre class="notranslate">Traceback (most recent call last):
  File <span class="pl-s"><span class="pl-pds">"</span>/Users/user/miniforge3/envs/clients/lib/python3.8/site-packages/IPython/core/interactiveshell.py<span class="pl-pds">"</span></span>, line 3433, <span class="pl-k">in</span> run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File <span class="pl-s"><span class="pl-pds">"</span>&lt;ipython-input-3-f76ea541d063&gt;<span class="pl-pds">"</span></span>, line 25, <span class="pl-k">in</span> <span class="pl-k">&lt;</span>module<span class="pl-k">&gt;</span>
    np.histogram(x, density=True, bins=<span class="pl-s"><span class="pl-pds">"</span>auto<span class="pl-pds">"</span></span>)
  File <span class="pl-s"><span class="pl-pds">"</span>&lt;__array_function__ internals&gt;<span class="pl-pds">"</span></span>, line 180, <span class="pl-k">in</span> histogram
  File <span class="pl-s"><span class="pl-pds">"</span>/Users/user/miniforge3/envs/clients/lib/python3.8/site-packages/numpy/lib/histograms.py<span class="pl-pds">"</span></span>, line 793, <span class="pl-k">in</span> histogram
    bin_edges, uniform_bins = _get_bin_edges(a, bins, range, weights)
  File <span class="pl-s"><span class="pl-pds">"</span>/Users/user/miniforge3/envs/clients/lib/python3.8/site-packages/numpy/lib/histograms.py<span class="pl-pds">"</span></span>, line 446, <span class="pl-k">in</span> _get_bin_edges
    bin_edges = np.linspace(
  File <span class="pl-s"><span class="pl-pds">"</span>&lt;__array_function__ internals&gt;<span class="pl-pds">"</span></span>, line 180, <span class="pl-k">in</span> linspace
  File <span class="pl-s"><span class="pl-pds">"</span>/Users/user/miniforge3/envs/clients/lib/python3.8/site-packages/numpy/core/function_base.py<span class="pl-pds">"</span></span>, line 135, <span class="pl-k">in</span> linspace
    y = _nx.arange(0, num, dtype=dt).reshape<span class="pl-s"><span class="pl-pds">((</span><span class="pl-k">-</span><span class="pl-c1">1</span><span class="pl-k">,</span>) <span class="pl-k">+</span> (<span class="pl-c1">1</span><span class="pl-k">,</span>) <span class="pl-k">*</span> ndim(delta<span class="pl-pds">))</span></span>
numpy.core._exceptions._ArrayMemoryError: Unable to allocate 12.7 PiB <span class="pl-k">for</span> an array with shape (1781062197026280,) and data <span class="pl-c1">type</span> float64</pre></div>
<h3 dir="auto">NumPy/Python version information:</h3>
<p dir="auto">1.23.3 3.8.15 | packaged by conda-forge | (default, Nov 22 2022, 08:49:06)<br>
[Clang 14.0.6 ]</p>
<h3 dir="auto">Context for the issue:</h3>
<p dir="auto"><em>No response</em></p>