<p dir="auto">Hello,</p>
<p dir="auto">I'm running into what I think is a bug in DataFrame.duplicated where it detects duplicates, but the data frame does not actually have any duplicated rows. It seems to only happen with integer columns, and somewhat large datasets (&gt;600,000 rows).</p>
<p dir="auto">I created a test data set to show the issue:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="df = pd.read_table(
  &quot;https://www.dropbox.com/s/vkw8bzxp290jitz/test.tab?raw=1&quot;,
  dtype = {&quot;chrom&quot; : &quot;int64&quot;,&quot;pos&quot; : &quot;int64&quot;}
)"><pre class="notranslate"><code class="notranslate">df = pd.read_table(
  "https://www.dropbox.com/s/vkw8bzxp290jitz/test.tab?raw=1",
  dtype = {"chrom" : "int64","pos" : "int64"}
)
</code></pre></div>
<p dir="auto">If you ask for duplicates, it will detect them:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="df.duplicated().any() # returns True"><pre class="notranslate"><code class="notranslate">df.duplicated().any() # returns True
</code></pre></div>
<p dir="auto">However, there are no duplicates:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="In [5]: from collections import Counter

In [6]: counter = Counter(zip(df.chrom,df.pos))

In [7]: counter.most_common(5)
Out[7]:
[((0, 13704091), 1),
 ((0, 201539008), 1),
 ((0, 8573433), 1),
 ((0, 127434927), 1),
 ((0, 247829766), 1)]"><pre class="notranslate"><code class="notranslate">In [5]: from collections import Counter

In [6]: counter = Counter(zip(df.chrom,df.pos))

In [7]: counter.most_common(5)
Out[7]:
[((0, 13704091), 1),
 ((0, 201539008), 1),
 ((0, 8573433), 1),
 ((0, 127434927), 1),
 ((0, 247829766), 1)]
</code></pre></div>
<p dir="auto">If I convert one of the columns to float, and then ask for duplicates, it is correct:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="df.loc[:,&quot;pos&quot;] = df.pos.astype(&quot;float64&quot;)
df.duplicated().any() # returns False"><pre class="notranslate"><code class="notranslate">df.loc[:,"pos"] = df.pos.astype("float64")
df.duplicated().any() # returns False
</code></pre></div>
<p dir="auto">Strangely, converting the first column <code class="notranslate">chrom</code> to float or string does not seem to matter.</p>
<p dir="auto">I had a difficult time in constructing this data frame to illustrate the example. It seems to only occur:</p>
<ul dir="auto">
<li>With at least two columns</li>
<li>One column (integer, string, float) that has a small number of unique values</li>
<li>One column (must be integer) that has a wide range of values, but in many instances, the values are close to each other</li>
<li>The dataset must be somewhat large, but it's hard to pin down. Somewhere on the order of &gt; 500,000 rows, it seems.</li>
</ul>
<p dir="auto">From looking quickly at the <code class="notranslate">DataFrame.duplicated</code> code, it looks like it is using a hash table of some kind, and using integer columns differently than other columns - perhaps it's ending up with collisions?</p>
<p dir="auto">Apologies if I'm missing something obvious here. Please let me know if I can be of any help in investigating further. My pandas version information is below.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="INSTALLED VERSIONS
------------------
commit: None
python: 2.7.10.final.0
python-bits: 64
OS: Darwin
OS-release: 14.5.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8

pandas: 0.17.0
nose: 1.3.7
pip: 7.1.2
setuptools: 18.4
Cython: 0.22.1
numpy: 1.10.1
scipy: 0.16.0
statsmodels: 0.6.1
IPython: 3.2.0
sphinx: 1.3.1
patsy: 0.3.0
dateutil: 2.4.2
pytz: 2015.6
blosc: None
bottleneck: None
tables: 3.2.0
numexpr: 2.4.3
matplotlib: 1.4.3
openpyxl: 2.0.2
xlrd: 0.9.4
xlwt: 1.0.0
xlsxwriter: 0.7.3
lxml: 3.4.4
bs4: 4.3.2
html5lib: None
httplib2: None
apiclient: None
sqlalchemy: 1.0.7
pymysql: None
psycopg2: None"><pre class="notranslate"><code class="notranslate">INSTALLED VERSIONS
------------------
commit: None
python: 2.7.10.final.0
python-bits: 64
OS: Darwin
OS-release: 14.5.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8

pandas: 0.17.0
nose: 1.3.7
pip: 7.1.2
setuptools: 18.4
Cython: 0.22.1
numpy: 1.10.1
scipy: 0.16.0
statsmodels: 0.6.1
IPython: 3.2.0
sphinx: 1.3.1
patsy: 0.3.0
dateutil: 2.4.2
pytz: 2015.6
blosc: None
bottleneck: None
tables: 3.2.0
numexpr: 2.4.3
matplotlib: 1.4.3
openpyxl: 2.0.2
xlrd: 0.9.4
xlwt: 1.0.0
xlsxwriter: 0.7.3
lxml: 3.4.4
bs4: 4.3.2
html5lib: None
httplib2: None
apiclient: None
sqlalchemy: 1.0.7
pymysql: None
psycopg2: None
</code></pre></div>
<p dir="auto">I appear to have some code that is sometimes (inexplicably) running in pure Python (i.e. <strong>not</strong> Cython-optimized). Specifically, the problem is with <code class="notranslate">loc[]</code> indexing on a boolean Series:</p>
<p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://cloud.githubusercontent.com/assets/9703332/10828484/e9995bbc-7e4a-11e5-9b00-ae367f328541.png"><img src="https://cloud.githubusercontent.com/assets/9703332/10828484/e9995bbc-7e4a-11e5-9b00-ae367f328541.png" alt="pandas slow loc lookup example" style="max-width: 100%;"></a></p>
<p dir="auto"><em>(Apologies for the redacted code and console output, but this project is not an open-source delivery)</em></p>
<p dir="auto">For context: <code class="notranslate">world.persons</code> returns a <code class="notranslate">DataFrame</code> with approximately 8 million records long and 15 columns wide. It uses a MultiIndex with 2 levels.</p>
<p dir="auto">This <code class="notranslate">run()</code> method is being called twice in the software sequence. The first time <code class="notranslate">run()</code> is called, <code class="notranslate">persons.loc[filter_column]</code> takes <strong>15 minutes</strong> to execute, while the second time it only takes 5 seconds.  What the heck is going on here?</p>
<p dir="auto">Furthermore, this is NOT the first time in the program that <code class="notranslate">world.persons</code> is being indexed, so I am not sure how caching is being applied.</p>
<p dir="auto">One other thing: prior to this <code class="notranslate">loc[]</code> call, my system starts to approach memory limits to the point that Windows begins paging.</p>
<p dir="auto">My OS is Windows 7 x64, and I'm using Pandas version 0.17.0.</p>