<p dir="auto">In below screenshot, I am scanning the database for a categorical called <code class="notranslate">classification_id</code> with the value <code class="notranslate">50ef44b795e6e42cd2000001</code> but I am getting a data-row where the categorical has the value 50ef44b795e6e42cd6000001`.</p>
<p dir="auto">How is this possible? Note that my list of categorical is huge, more than 4 million entries, with 12 million total rows. (Yes, on average, each classification_id appears 3 times.)</p>
<p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://cloud.githubusercontent.com/assets/69774/10439330/7755e45a-70f6-11e5-9dc6-26ee7359d95a.png"><img width="931" alt="screenshot 2015-10-12 15 31 25" src="https://cloud.githubusercontent.com/assets/69774/10439330/7755e45a-70f6-11e5-9dc6-26ee7359d95a.png" style="max-width: 100%;"></a></p>
<p dir="auto">On a side note: The display of this one row of numpy in the line with <code class="notranslate">.values</code> at the end takes a lot of time, possibly due to large size of the Categorical, can that be avoided somehow?</p>
<p dir="auto">Here's my required meta-data for the bug report:<br>
pandas Version: 0.17.0</p>
<h2 dir="auto">INSTALLED VERSIONS</h2>
<p dir="auto">commit: None<br>
python: 3.4.3.final.0<br>
python-bits: 64<br>
OS: Darwin<br>
OS-release: 14.5.0<br>
machine: x86_64<br>
processor: i386<br>
byteorder: little<br>
LC_ALL: None<br>
LANG: en_US.UTF-8</p>
<p dir="auto">pandas: 0.17.0<br>
nose: 1.3.7<br>
pip: 7.1.2<br>
setuptools: 18.3.2<br>
Cython: None<br>
numpy: 1.10.0<br>
scipy: 0.16.0<br>
statsmodels: None<br>
IPython: 4.1.0-dev<br>
sphinx: None<br>
patsy: None<br>
dateutil: 2.4.2<br>
pytz: 2015.6<br>
blosc: None<br>
bottleneck: None<br>
tables: 3.2.2<br>
numexpr: 2.4.4<br>
matplotlib: 1.4.3<br>
openpyxl: None<br>
xlrd: None<br>
xlwt: None<br>
xlsxwriter: None<br>
lxml: None<br>
bs4: None<br>
html5lib: None<br>
httplib2: None<br>
apiclient: None<br>
sqlalchemy: 1.0.8<br>
pymysql: None<br>
psycopg2: None</p>
<h4 dir="auto">Code Sample, a copy-pastable example if possible</h4>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="# Your code here
cdef inline void remove_var(double val, double *nobs, double *mean_x,
                            double *ssqdm_x) nogil:
    &quot;&quot;&quot; remove a value from the var calc &quot;&quot;&quot;
    cdef double delta

    # Not NaN
    if val == val:
        nobs[0] = nobs[0] - 1
        if nobs[0]:
            # a part of Welford's method for the online variance-calculation
            # https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance
            delta = val - mean_x[0]
            mean_x[0] = mean_x[0] - delta / nobs[0]
            ssqdm_x[0] = ssqdm_x[0] - ((nobs[0] + 1) * delta ** 2) / nobs[0]
        else:
            mean_x[0] = 0
ssqdm_x[0] = 0


nobs is a pointer to a double and is incremented and decrement accordingly as add_var and remove_var are called. if nobs == 0, then you want to reset mean_x and ssqdm_x to 0 as you do in the else condition. my question is this -- isn't it possible that nobs could be some epsilon value and not exactly equal to 0.0 ? I dont know floating point math that well, and it could be fine since all we do is incrememt by1 or decrement by 1 so it could be fine. just wondering in case this should be:
if abs(nbos[0]) &lt; 1e-11 or something

thanks

"><pre class="notranslate"><span class="pl-c"># Your code here</span>
<span class="pl-s1">cdef</span> <span class="pl-s1">inline</span> <span class="pl-s1">void</span> <span class="pl-en">remove_var</span>(<span class="pl-s1">double</span> <span class="pl-s1">val</span>, <span class="pl-s1">double</span> <span class="pl-c1">*</span><span class="pl-s1">nobs</span>, <span class="pl-s1">double</span> <span class="pl-c1">*</span><span class="pl-s1">mean_x</span>,
                            <span class="pl-s1">double</span> <span class="pl-c1">*</span><span class="pl-s1">ssqdm_x</span>) <span class="pl-s1">nogil</span>:
    <span class="pl-s">""" remove a value from the var calc """</span>
    <span class="pl-s1">cdef</span> <span class="pl-s1">double</span> <span class="pl-s1">delta</span>

    <span class="pl-c"># Not NaN</span>
    <span class="pl-k">if</span> <span class="pl-s1">val</span> <span class="pl-c1">==</span> <span class="pl-s1">val</span>:
        <span class="pl-s1">nobs</span>[<span class="pl-c1">0</span>] <span class="pl-c1">=</span> <span class="pl-s1">nobs</span>[<span class="pl-c1">0</span>] <span class="pl-c1">-</span> <span class="pl-c1">1</span>
        <span class="pl-k">if</span> <span class="pl-s1">nobs</span>[<span class="pl-c1">0</span>]:
            <span class="pl-c"># a part of Welford's method for the online variance-calculation</span>
            <span class="pl-c"># https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance</span>
            <span class="pl-s1">delta</span> <span class="pl-c1">=</span> <span class="pl-s1">val</span> <span class="pl-c1">-</span> <span class="pl-s1">mean_x</span>[<span class="pl-c1">0</span>]
            <span class="pl-s1">mean_x</span>[<span class="pl-c1">0</span>] <span class="pl-c1">=</span> <span class="pl-s1">mean_x</span>[<span class="pl-c1">0</span>] <span class="pl-c1">-</span> <span class="pl-s1">delta</span> <span class="pl-c1">/</span> <span class="pl-s1">nobs</span>[<span class="pl-c1">0</span>]
            <span class="pl-s1">ssqdm_x</span>[<span class="pl-c1">0</span>] <span class="pl-c1">=</span> <span class="pl-s1">ssqdm_x</span>[<span class="pl-c1">0</span>] <span class="pl-c1">-</span> ((<span class="pl-s1">nobs</span>[<span class="pl-c1">0</span>] <span class="pl-c1">+</span> <span class="pl-c1">1</span>) <span class="pl-c1">*</span> <span class="pl-s1">delta</span> <span class="pl-c1">**</span> <span class="pl-c1">2</span>) <span class="pl-c1">/</span> <span class="pl-s1">nobs</span>[<span class="pl-c1">0</span>]
        <span class="pl-k">else</span>:
            <span class="pl-s1">mean_x</span>[<span class="pl-c1">0</span>] <span class="pl-c1">=</span> <span class="pl-c1">0</span>
<span class="pl-s1">ssqdm_x</span>[<span class="pl-c1">0</span>] <span class="pl-c1">=</span> <span class="pl-c1">0</span>


<span class="pl-s1">nobs</span> <span class="pl-c1">is</span> <span class="pl-s1">a</span> <span class="pl-s1">pointer</span> <span class="pl-s1">to</span> <span class="pl-s1">a</span> <span class="pl-s1">double</span> <span class="pl-c1">and</span> <span class="pl-s1">is</span> <span class="pl-s1">incremented</span> <span class="pl-c1">and</span> <span class="pl-s1">decrement</span> <span class="pl-s1">accordingly</span> <span class="pl-k">as</span> <span class="pl-s1">add_var</span> <span class="pl-c1">and</span> <span class="pl-s1">remove_var</span> <span class="pl-s1">are</span> <span class="pl-s1">called</span>. <span class="pl-s1">if</span> <span class="pl-s1">nobs</span> <span class="pl-c1">==</span> <span class="pl-c1">0</span>, <span class="pl-s1">then</span> <span class="pl-s1">you</span> <span class="pl-s1">want</span> <span class="pl-s1">to</span> <span class="pl-s1">reset</span> <span class="pl-s1">mean_x</span> <span class="pl-c1">and</span> <span class="pl-s1">ssqdm_x</span> <span class="pl-s1">to</span> <span class="pl-c1">0</span> <span class="pl-k">as</span> <span class="pl-s1">you</span> <span class="pl-s1">do</span> <span class="pl-c1">in</span> <span class="pl-s1">the</span> <span class="pl-s1">else</span> <span class="pl-s1">condition</span>. <span class="pl-s1">my</span> <span class="pl-s1">question</span> <span class="pl-c1">is</span> <span class="pl-s1">this</span> <span class="pl-c1">-</span><span class="pl-c1">-</span> <span class="pl-s1">isn</span>'<span class="pl-s1">t</span> <span class="pl-s1">it</span> <span class="pl-s1">possible</span> <span class="pl-s1">that</span> <span class="pl-s1">nobs</span> <span class="pl-s1">could</span> <span class="pl-s1">be</span> <span class="pl-s1">some</span> <span class="pl-s1">epsilon</span> <span class="pl-s1">value</span> <span class="pl-c1">and</span> <span class="pl-c1">not</span> <span class="pl-s1">exactly</span> <span class="pl-s1">equal</span> <span class="pl-s1">to</span> <span class="pl-c1">0.0</span> ? <span class="pl-v">I</span> <span class="pl-s1">dont</span> <span class="pl-s1">know</span> <span class="pl-s1">floating</span> <span class="pl-s1">point</span> <span class="pl-s1">math</span> <span class="pl-s1">that</span> <span class="pl-s1">well</span>, <span class="pl-s1">and</span> <span class="pl-s1">it</span> <span class="pl-s1">could</span> <span class="pl-s1">be</span> <span class="pl-s1">fine</span> <span class="pl-s1">since</span> <span class="pl-s1">all</span> <span class="pl-s1">we</span> <span class="pl-s1">do</span> <span class="pl-c1">is</span> <span class="pl-s1">incrememt</span> <span class="pl-s1">by1</span> <span class="pl-c1">or</span> <span class="pl-s1">decrement</span> <span class="pl-s1">by</span> <span class="pl-c1">1</span> <span class="pl-s1">so</span> <span class="pl-s1">it</span> <span class="pl-s1">could</span> <span class="pl-s1">be</span> <span class="pl-s1">fine</span>. <span class="pl-s1">just</span> <span class="pl-s1">wondering</span> <span class="pl-c1">in</span> <span class="pl-s1">case</span> <span class="pl-s1">this</span> <span class="pl-s1">should</span> <span class="pl-s1">be</span>:
<span class="pl-s1">if</span> <span class="pl-en">abs</span>(<span class="pl-s1">nbos</span>[<span class="pl-c1">0</span>]) <span class="pl-c1">&lt;</span> <span class="pl-c1">1e-11</span> <span class="pl-c1">or</span> <span class="pl-s1">something</span>

<span class="pl-s1">thanks</span>
</pre></div>
<h4 dir="auto">Problem description</h4>
<p dir="auto">[this should explain <strong>why</strong> the current behaviour is a problem and why the expected output is a better solution.]</p>
<p dir="auto"><strong>Note</strong>: We receive a lot of issues on our GitHub tracker, so it is very possible that your issue has been posted before. Please check first before submitting so that we do not have to handle and close duplicates!</p>
<p dir="auto"><strong>Note</strong>: Many problems can be resolved by simply upgrading <code class="notranslate">pandas</code> to the latest version. Before submitting, please check if that solution works for you. If possible, you may want to check if <code class="notranslate">master</code> addresses this issue, but that is not necessary.</p>
<p dir="auto">For documentation-related issues, you can check the latest versions of the docs on <code class="notranslate">master</code> here:</p>
<p dir="auto"><a href="https://pandas-docs.github.io/pandas-docs-travis/" rel="nofollow">https://pandas-docs.github.io/pandas-docs-travis/</a></p>
<p dir="auto">If the issue has not been resolved there, go ahead and file it in the issue tracker.</p>
<h4 dir="auto">Expected Output</h4>
<h4 dir="auto">Output of <code class="notranslate">pd.show_versions()</code></h4>
<details>
<p dir="auto">[paste the output of <code class="notranslate">pd.show_versions()</code> here below this line]</p>
</details>