<p dir="auto">xref <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="215316030" data-permission-text="Title is private" data-url="https://github.com/pandas-dev/pandas/issues/15738" data-hovercard-type="pull_request" data-hovercard-url="/pandas-dev/pandas/pull/15738/hovercard?comment_id=287843016&amp;comment_type=issue_comment" href="https://github.com/pandas-dev/pandas/pull/15738#issuecomment-287843016">#15738 (comment)</a></p>
<p dir="auto">Currently bool data is passed to the generic python object hashtable for the following methods, making them all very slow.</p>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox">  <code class="notranslate">value_counts</code></li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox">  <code class="notranslate">rank</code></li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox">  <code class="notranslate">unique</code></li>
</ul>
<p dir="auto">Linked PR casts to int for <code class="notranslate">factorize</code>, <code class="notranslate">duplicated</code>, <code class="notranslate">drop_duplicates</code></p>
<p dir="auto">We could skip the hashing altogether and take advantage of the fixed set of values, e.g. below is a fastpath for unique.</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="%%cython
from numpy cimport uint8_t
cimport cython

import numpy as np

@cython.boundscheck(False)
@cython.wraparound(False)
def unique(uint8_t[:] data):
    cdef:
        bint seen_true = False
        bint seen_false = False
        Py_ssize_t i, N = len(data)
        uint8_t val
    with nogil:
        for i in range(N):
            val = data[i]
            if val == 0:
                seen_false = True
            elif val == 1:
                seen_true = True
            if seen_true and seen_false:
                break
    if seen_true and seen_false:
        return np.array([True, False])
    elif seen_true:
        return np.array([True])
    elif seen_false:
        return np.array([False])
    else:
        return np.array([], dtype=bool)

In [35]: bools = np.array([False, True] * 100000)

In [36]: %timeit pd.unique(bools.view('uint8')).astype(bool)
1000 loops, best of 3: 1.74 ms per loop

In [37]: %timeit unique(bools.view('uint8'))
100000 loops, best of 3: 3.47 µs per loop"><pre class="notranslate"><span class="pl-c1">%</span><span class="pl-c1">%</span><span class="pl-s1">cython</span>
<span class="pl-k">from</span> <span class="pl-s1">numpy</span> <span class="pl-s1">cimport</span> <span class="pl-s1">uint8_t</span>
<span class="pl-s1">cimport</span> <span class="pl-s1">cython</span>

<span class="pl-k">import</span> <span class="pl-s1">numpy</span> <span class="pl-k">as</span> <span class="pl-s1">np</span>

<span class="pl-en">@<span class="pl-s1">cython</span>.<span class="pl-en">boundscheck</span>(<span class="pl-c1">False</span>)</span>
<span class="pl-en">@<span class="pl-s1">cython</span>.<span class="pl-en">wraparound</span>(<span class="pl-c1">False</span>)</span>
<span class="pl-k">def</span> <span class="pl-en">unique</span>(<span class="pl-s1">uint8_t</span>[:] <span class="pl-s1">data</span>):
    <span class="pl-s1">cdef</span>:
        <span class="pl-s1">bint</span> <span class="pl-s1">seen_true</span> <span class="pl-c1">=</span> <span class="pl-c1">False</span>
        <span class="pl-s1">bint</span> <span class="pl-s1">seen_false</span> <span class="pl-c1">=</span> <span class="pl-c1">False</span>
        <span class="pl-v">Py_ssize_t</span> <span class="pl-s1">i</span>, <span class="pl-v">N</span> <span class="pl-c1">=</span> <span class="pl-en">len</span>(<span class="pl-s1">data</span>)
        <span class="pl-s1">uint8_t</span> <span class="pl-s1">val</span>
    <span class="pl-k">with</span> <span class="pl-s1">nogil</span>:
        <span class="pl-k">for</span> <span class="pl-s1">i</span> <span class="pl-c1">in</span> <span class="pl-en">range</span>(<span class="pl-v">N</span>):
            <span class="pl-s1">val</span> <span class="pl-c1">=</span> <span class="pl-s1">data</span>[<span class="pl-s1">i</span>]
            <span class="pl-k">if</span> <span class="pl-s1">val</span> <span class="pl-c1">==</span> <span class="pl-c1">0</span>:
                <span class="pl-s1">seen_false</span> <span class="pl-c1">=</span> <span class="pl-c1">True</span>
            <span class="pl-k">elif</span> <span class="pl-s1">val</span> <span class="pl-c1">==</span> <span class="pl-c1">1</span>:
                <span class="pl-s1">seen_true</span> <span class="pl-c1">=</span> <span class="pl-c1">True</span>
            <span class="pl-k">if</span> <span class="pl-s1">seen_true</span> <span class="pl-c1">and</span> <span class="pl-s1">seen_false</span>:
                <span class="pl-k">break</span>
    <span class="pl-k">if</span> <span class="pl-s1">seen_true</span> <span class="pl-c1">and</span> <span class="pl-s1">seen_false</span>:
        <span class="pl-k">return</span> <span class="pl-s1">np</span>.<span class="pl-en">array</span>([<span class="pl-c1">True</span>, <span class="pl-c1">False</span>])
    <span class="pl-k">elif</span> <span class="pl-s1">seen_true</span>:
        <span class="pl-k">return</span> <span class="pl-s1">np</span>.<span class="pl-en">array</span>([<span class="pl-c1">True</span>])
    <span class="pl-k">elif</span> <span class="pl-s1">seen_false</span>:
        <span class="pl-k">return</span> <span class="pl-s1">np</span>.<span class="pl-en">array</span>([<span class="pl-c1">False</span>])
    <span class="pl-k">else</span>:
        <span class="pl-k">return</span> <span class="pl-s1">np</span>.<span class="pl-en">array</span>([], <span class="pl-s1">dtype</span><span class="pl-c1">=</span><span class="pl-s1">bool</span>)

<span class="pl-v">In</span> [<span class="pl-c1">35</span>]: <span class="pl-s1">bools</span> <span class="pl-c1">=</span> <span class="pl-s1">np</span>.<span class="pl-en">array</span>([<span class="pl-c1">False</span>, <span class="pl-c1">True</span>] <span class="pl-c1">*</span> <span class="pl-c1">100000</span>)

<span class="pl-v">In</span> [<span class="pl-c1">36</span>]: <span class="pl-c1">%</span><span class="pl-s1">timeit</span> <span class="pl-s1">pd</span>.<span class="pl-en">unique</span>(<span class="pl-s1">bools</span>.<span class="pl-en">view</span>(<span class="pl-s">'uint8'</span>)).<span class="pl-en">astype</span>(<span class="pl-s1">bool</span>)
<span class="pl-c1">1000</span> <span class="pl-s1">loops</span>, <span class="pl-s1">best</span> <span class="pl-s1">of</span> <span class="pl-c1">3</span>: <span class="pl-c1">1.74</span> <span class="pl-s1">ms</span> <span class="pl-s1">per</span> <span class="pl-s1">loop</span>

<span class="pl-v">In</span> [<span class="pl-c1">37</span>]: <span class="pl-c1">%</span><span class="pl-s1">timeit</span> <span class="pl-en">unique</span>(<span class="pl-s1">bools</span>.<span class="pl-en">view</span>(<span class="pl-s">'uint8'</span>))
<span class="pl-c1">100000</span> <span class="pl-s1">loops</span>, <span class="pl-s1">best</span> <span class="pl-s1">of</span> <span class="pl-c1">3</span>: <span class="pl-c1">3.47</span> <span class="pl-s1">µs</span> <span class="pl-s1">per</span> <span class="pl-s1">loop</span></pre></div>
<h4 dir="auto">Code Sample, a copy-pastable example if possible</h4>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import pandas as pd
import numpy as np

s = pd.Series(np.random.randint(0, 1000000, size=100000).astype('datetime64[ns]')).sort_values()
s[s.duplicated()].head()"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">pandas</span> <span class="pl-k">as</span> <span class="pl-s1">pd</span>
<span class="pl-k">import</span> <span class="pl-s1">numpy</span> <span class="pl-k">as</span> <span class="pl-s1">np</span>

<span class="pl-s1">s</span> <span class="pl-c1">=</span> <span class="pl-s1">pd</span>.<span class="pl-v">Series</span>(<span class="pl-s1">np</span>.<span class="pl-s1">random</span>.<span class="pl-en">randint</span>(<span class="pl-c1">0</span>, <span class="pl-c1">1000000</span>, <span class="pl-s1">size</span><span class="pl-c1">=</span><span class="pl-c1">100000</span>).<span class="pl-en">astype</span>(<span class="pl-s">'datetime64[ns]'</span>)).<span class="pl-en">sort_values</span>()
<span class="pl-s1">s</span>[<span class="pl-s1">s</span>.<span class="pl-en">duplicated</span>()].<span class="pl-en">head</span>()</pre></div>
<h4 dir="auto">Problem description</h4>
<p dir="auto">Random output on my system:</p>
<p dir="auto">70570   1970-01-01 00:00:00.000000159<br>
90556   1970-01-01 00:00:00.000000248<br>
28781   1970-01-01 00:00:00.000000274<br>
73616   1970-01-01 00:00:00.000000276<br>
31370   1970-01-01 00:00:00.000000409</p>
<p dir="auto">I am not sure why I see duplicates for sub-second values. Adding small random nanosecond deltas seems like a perfect way to de-dupe an index. it does not seem to work though.</p>
<h4 dir="auto">Expected Output</h4>
<p dir="auto">Empty.</p>
<h4 dir="auto">Output of <code class="notranslate">pd.show_versions()</code></h4>
<h2 dir="auto">INSTALLED VERSIONS</h2>
<p dir="auto">commit: None<br>
python: 2.7.11.final.0<br>
python-bits: 64<br>
OS: Darwin<br>
OS-release: 16.7.0<br>
machine: x86_64<br>
processor: i386<br>
byteorder: little<br>
LC_ALL: None<br>
LANG: en_US.UTF-8<br>
LOCALE: None.None</p>
<p dir="auto">pandas: 0.20.3<br>
pytest: None<br>
pip: 9.0.1<br>
setuptools: 18.7.1<br>
Cython: None<br>
numpy: 1.13.1<br>
scipy: 0.15.1<br>
xarray: None<br>
IPython: 3.0.0<br>
sphinx: None<br>
patsy: None<br>
dateutil: 2.6.1<br>
pytz: 2017.2<br>
blosc: None<br>
bottleneck: None<br>
tables: None<br>
numexpr: None<br>
feather: None<br>
matplotlib: 1.4.3<br>
openpyxl: None<br>
xlrd: None<br>
xlwt: None<br>
xlsxwriter: None<br>
lxml: None<br>
bs4: None<br>
html5lib: None<br>
sqlalchemy: None<br>
pymysql: None<br>
psycopg2: None<br>
jinja2: 2.7.3<br>
s3fs: None<br>
pandas_gbq: None<br>
pandas_datareader: None</p>