<p dir="auto">I wish I could say more about this, but it is with proprietary data.  Basically, I have a DataFrame of 10 columns and around 7000 rows.  When I call "duplicated" on 4 of the rows (3 have strings, 1 has a float), I get around 10 items being flagged as duplicate (5 pairs) when in fact, they are different.</p>
<p dir="auto">This is sample text of what is being flagged:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="          Type   Line                                         LineString  \
1885      else  832.0                (&amp;temp32)-&gt;byte1 = (&amp;temp32)-&gt;byte4   
1895  do while  832.0                                        do while(0)   
4515      enum  122.0    enum {QQ_ON_UNASSERTED = 0, QQ_ON_ASSERTED = 1}   
4521      enum  167.0          enum {FIELD_RESET = 1, FIELD_TRIPPED = 0}   

              Parameter  Filename  
1885             temp32   arinc.c  
1895                      arinc.c  
4515      QQ_ON_ASSERTED  eeprom.c  
4521       FIELD_TRIPPED  eeprom.c  "><pre class="notranslate"><code class="notranslate">          Type   Line                                         LineString  \
1885      else  832.0                (&amp;temp32)-&gt;byte1 = (&amp;temp32)-&gt;byte4   
1895  do while  832.0                                        do while(0)   
4515      enum  122.0    enum {QQ_ON_UNASSERTED = 0, QQ_ON_ASSERTED = 1}   
4521      enum  167.0          enum {FIELD_RESET = 1, FIELD_TRIPPED = 0}   

              Parameter  Filename  
1885             temp32   arinc.c  
1895                      arinc.c  
4515      QQ_ON_ASSERTED  eeprom.c  
4521       FIELD_TRIPPED  eeprom.c  
</code></pre></div>
<p dir="auto">I know that duplicates are checked through hashing, but is there some way to, I don't know, compare a checksum or a more robust measurement to ensure that only duplicates are flagged?  What is the chance that two items could be hashed to the same value?</p>
<p dir="auto">By adding more of the fields to check the data I am able to prevent false duplicate flags.</p>
<p dir="auto">I am using Pandas 0.18 from WinPython-64bit-3.4.4.2Qt5.</p>
<h4 dir="auto">Code Sample, a copy-pastable example if possible</h4>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="# Your code here
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from matplotlib import dates

t = np.arange(10)
ts = pd.datetime(2017, 11, 16) + pd.to_timedelta(t, 's')
df = pd.DataFrame(t, index=ts)

fig, ax = plt.subplots()
ax.plot(df)
# workaround:
# ax.plot(df.index.to_pydatetime(), df)

# format time axis
hfmt = dates.DateFormatter('%m/%d %H:%M')
ax.xaxis.set_major_formatter(hfmt)
fig.autofmt_xdate()
plt.show()"><pre class="notranslate"><span class="pl-c"># Your code here</span>
<span class="pl-k">import</span> <span class="pl-s1">numpy</span> <span class="pl-k">as</span> <span class="pl-s1">np</span>
<span class="pl-k">import</span> <span class="pl-s1">pandas</span> <span class="pl-k">as</span> <span class="pl-s1">pd</span>
<span class="pl-k">import</span> <span class="pl-s1">matplotlib</span>.<span class="pl-s1">pyplot</span> <span class="pl-k">as</span> <span class="pl-s1">plt</span>
<span class="pl-k">from</span> <span class="pl-s1">matplotlib</span> <span class="pl-k">import</span> <span class="pl-s1">dates</span>

<span class="pl-s1">t</span> <span class="pl-c1">=</span> <span class="pl-s1">np</span>.<span class="pl-en">arange</span>(<span class="pl-c1">10</span>)
<span class="pl-s1">ts</span> <span class="pl-c1">=</span> <span class="pl-s1">pd</span>.<span class="pl-en">datetime</span>(<span class="pl-c1">2017</span>, <span class="pl-c1">11</span>, <span class="pl-c1">16</span>) <span class="pl-c1">+</span> <span class="pl-s1">pd</span>.<span class="pl-en">to_timedelta</span>(<span class="pl-s1">t</span>, <span class="pl-s">'s'</span>)
<span class="pl-s1">df</span> <span class="pl-c1">=</span> <span class="pl-s1">pd</span>.<span class="pl-v">DataFrame</span>(<span class="pl-s1">t</span>, <span class="pl-s1">index</span><span class="pl-c1">=</span><span class="pl-s1">ts</span>)

<span class="pl-s1">fig</span>, <span class="pl-s1">ax</span> <span class="pl-c1">=</span> <span class="pl-s1">plt</span>.<span class="pl-en">subplots</span>()
<span class="pl-s1">ax</span>.<span class="pl-en">plot</span>(<span class="pl-s1">df</span>)
<span class="pl-c"># workaround:</span>
<span class="pl-c"># ax.plot(df.index.to_pydatetime(), df)</span>

<span class="pl-c"># format time axis</span>
<span class="pl-s1">hfmt</span> <span class="pl-c1">=</span> <span class="pl-s1">dates</span>.<span class="pl-v">DateFormatter</span>(<span class="pl-s">'%m/%d %H:%M'</span>)
<span class="pl-s1">ax</span>.<span class="pl-s1">xaxis</span>.<span class="pl-en">set_major_formatter</span>(<span class="pl-s1">hfmt</span>)
<span class="pl-s1">fig</span>.<span class="pl-en">autofmt_xdate</span>()
<span class="pl-s1">plt</span>.<span class="pl-en">show</span>()</pre></div>
<h4 dir="auto">Problem description</h4>
<p dir="auto">Version: <code class="notranslate">pandas-0.21.0</code></p>
<p dir="auto">trace:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Traceback (most recent call last):

  File &quot;D:\Anaconda3\lib\site-packages\IPython\core\formatters.py&quot;, line 332, in __call__
    return printer(obj)

  File &quot;D:\Anaconda3\lib\site-packages\IPython\core\pylabtools.py&quot;, line 237, in &lt;lambda&gt;
    png_formatter.for_type(Figure, lambda fig: print_figure(fig, 'png', **kwargs))

  File &quot;D:\Anaconda3\lib\site-packages\IPython\core\pylabtools.py&quot;, line 121, in print_figure
    fig.canvas.print_figure(bytes_io, **kw)

  File &quot;D:\Anaconda3\lib\site-packages\matplotlib\backend_bases.py&quot;, line 2208, in print_figure
    **kwargs)

  File &quot;D:\Anaconda3\lib\site-packages\matplotlib\backends\backend_agg.py&quot;, line 507, in print_png
    FigureCanvasAgg.draw(self)

  File &quot;D:\Anaconda3\lib\site-packages\matplotlib\backends\backend_agg.py&quot;, line 430, in draw
    self.figure.draw(self.renderer)

  File &quot;D:\Anaconda3\lib\site-packages\matplotlib\artist.py&quot;, line 55, in draw_wrapper
    return draw(artist, renderer, *args, **kwargs)

  File &quot;D:\Anaconda3\lib\site-packages\matplotlib\figure.py&quot;, line 1295, in draw
    renderer, self, artists, self.suppressComposite)

  File &quot;D:\Anaconda3\lib\site-packages\matplotlib\image.py&quot;, line 138, in _draw_list_compositing_images
    a.draw(renderer)

  File &quot;D:\Anaconda3\lib\site-packages\matplotlib\artist.py&quot;, line 55, in draw_wrapper
    return draw(artist, renderer, *args, **kwargs)

  File &quot;D:\Anaconda3\lib\site-packages\matplotlib\axes\_base.py&quot;, line 2399, in draw
    mimage._draw_list_compositing_images(renderer, self, artists)

  File &quot;D:\Anaconda3\lib\site-packages\matplotlib\image.py&quot;, line 138, in _draw_list_compositing_images
    a.draw(renderer)

  File &quot;D:\Anaconda3\lib\site-packages\matplotlib\artist.py&quot;, line 55, in draw_wrapper
    return draw(artist, renderer, *args, **kwargs)

  File &quot;D:\Anaconda3\lib\site-packages\matplotlib\axis.py&quot;, line 1133, in draw
    ticks_to_draw = self._update_ticks(renderer)

  File &quot;D:\Anaconda3\lib\site-packages\matplotlib\axis.py&quot;, line 974, in _update_ticks
    tick_tups = list(self.iter_ticks())

  File &quot;D:\Anaconda3\lib\site-packages\matplotlib\axis.py&quot;, line 921, in iter_ticks
    for i, val in enumerate(majorLocs)]

  File &quot;D:\Anaconda3\lib\site-packages\matplotlib\axis.py&quot;, line 921, in &lt;listcomp&gt;
    for i, val in enumerate(majorLocs)]

  File &quot;D:\Anaconda3\lib\site-packages\matplotlib\dates.py&quot;, line 538, in __call__
    dt = num2date(x, self.tz)

  File &quot;D:\Anaconda3\lib\site-packages\matplotlib\dates.py&quot;, line 441, in num2date
    return _from_ordinalf(x, tz)

  File &quot;D:\Anaconda3\lib\site-packages\matplotlib\dates.py&quot;, line 256, in _from_ordinalf
    dt = datetime.datetime.fromordinal(ix).replace(tzinfo=UTC)

OverflowError: Python int too large to convert to C long

&lt;matplotlib.figure.Figure at 0xebecc88&gt;"><pre class="notranslate"><code class="notranslate">Traceback (most recent call last):

  File "D:\Anaconda3\lib\site-packages\IPython\core\formatters.py", line 332, in __call__
    return printer(obj)

  File "D:\Anaconda3\lib\site-packages\IPython\core\pylabtools.py", line 237, in &lt;lambda&gt;
    png_formatter.for_type(Figure, lambda fig: print_figure(fig, 'png', **kwargs))

  File "D:\Anaconda3\lib\site-packages\IPython\core\pylabtools.py", line 121, in print_figure
    fig.canvas.print_figure(bytes_io, **kw)

  File "D:\Anaconda3\lib\site-packages\matplotlib\backend_bases.py", line 2208, in print_figure
    **kwargs)

  File "D:\Anaconda3\lib\site-packages\matplotlib\backends\backend_agg.py", line 507, in print_png
    FigureCanvasAgg.draw(self)

  File "D:\Anaconda3\lib\site-packages\matplotlib\backends\backend_agg.py", line 430, in draw
    self.figure.draw(self.renderer)

  File "D:\Anaconda3\lib\site-packages\matplotlib\artist.py", line 55, in draw_wrapper
    return draw(artist, renderer, *args, **kwargs)

  File "D:\Anaconda3\lib\site-packages\matplotlib\figure.py", line 1295, in draw
    renderer, self, artists, self.suppressComposite)

  File "D:\Anaconda3\lib\site-packages\matplotlib\image.py", line 138, in _draw_list_compositing_images
    a.draw(renderer)

  File "D:\Anaconda3\lib\site-packages\matplotlib\artist.py", line 55, in draw_wrapper
    return draw(artist, renderer, *args, **kwargs)

  File "D:\Anaconda3\lib\site-packages\matplotlib\axes\_base.py", line 2399, in draw
    mimage._draw_list_compositing_images(renderer, self, artists)

  File "D:\Anaconda3\lib\site-packages\matplotlib\image.py", line 138, in _draw_list_compositing_images
    a.draw(renderer)

  File "D:\Anaconda3\lib\site-packages\matplotlib\artist.py", line 55, in draw_wrapper
    return draw(artist, renderer, *args, **kwargs)

  File "D:\Anaconda3\lib\site-packages\matplotlib\axis.py", line 1133, in draw
    ticks_to_draw = self._update_ticks(renderer)

  File "D:\Anaconda3\lib\site-packages\matplotlib\axis.py", line 974, in _update_ticks
    tick_tups = list(self.iter_ticks())

  File "D:\Anaconda3\lib\site-packages\matplotlib\axis.py", line 921, in iter_ticks
    for i, val in enumerate(majorLocs)]

  File "D:\Anaconda3\lib\site-packages\matplotlib\axis.py", line 921, in &lt;listcomp&gt;
    for i, val in enumerate(majorLocs)]

  File "D:\Anaconda3\lib\site-packages\matplotlib\dates.py", line 538, in __call__
    dt = num2date(x, self.tz)

  File "D:\Anaconda3\lib\site-packages\matplotlib\dates.py", line 441, in num2date
    return _from_ordinalf(x, tz)

  File "D:\Anaconda3\lib\site-packages\matplotlib\dates.py", line 256, in _from_ordinalf
    dt = datetime.datetime.fromordinal(ix).replace(tzinfo=UTC)

OverflowError: Python int too large to convert to C long

&lt;matplotlib.figure.Figure at 0xebecc88&gt;
</code></pre></div>
<h4 dir="auto">Expected Output</h4>
<h4 dir="auto">Output of <code class="notranslate">pd.show_versions()</code></h4>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="INSTALLED VERSIONS
------------------
commit: None
python: 3.6.3.final.0
python-bits: 64
OS: Windows
OS-release: 7
machine: AMD64
processor: Intel64 Family 6 Model 60 Stepping 3, GenuineIntel
byteorder: little
LC_ALL: None
LANG: en
LOCALE: None.None

pandas: 0.21.0
pytest: 3.2.1
pip: 9.0.1
setuptools: 36.5.0.post20170921
Cython: 0.26.1
numpy: 1.13.3
scipy: 1.0.0
pyarrow: None
xarray: 0.9.6
IPython: 6.1.0
sphinx: 1.6.3
patsy: 0.4.1
dateutil: 2.6.1
pytz: 2017.2
blosc: None
bottleneck: 1.2.1
tables: 3.4.2
numexpr: 2.6.2
feather: None
matplotlib: 2.1.0
openpyxl: 2.4.8
xlrd: 1.1.0
xlwt: 1.3.0
xlsxwriter: 1.0.2
lxml: 4.1.0
bs4: 4.6.0
html5lib: 0.999999999
sqlalchemy: 1.1.13
pymysql: None
psycopg2: None
jinja2: 2.9.6
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None"><pre class="notranslate"><code class="notranslate">INSTALLED VERSIONS
------------------
commit: None
python: 3.6.3.final.0
python-bits: 64
OS: Windows
OS-release: 7
machine: AMD64
processor: Intel64 Family 6 Model 60 Stepping 3, GenuineIntel
byteorder: little
LC_ALL: None
LANG: en
LOCALE: None.None

pandas: 0.21.0
pytest: 3.2.1
pip: 9.0.1
setuptools: 36.5.0.post20170921
Cython: 0.26.1
numpy: 1.13.3
scipy: 1.0.0
pyarrow: None
xarray: 0.9.6
IPython: 6.1.0
sphinx: 1.6.3
patsy: 0.4.1
dateutil: 2.6.1
pytz: 2017.2
blosc: None
bottleneck: 1.2.1
tables: 3.4.2
numexpr: 2.6.2
feather: None
matplotlib: 2.1.0
openpyxl: 2.4.8
xlrd: 1.1.0
xlwt: 1.3.0
xlsxwriter: 1.0.2
lxml: 4.1.0
bs4: 4.6.0
html5lib: 0.999999999
sqlalchemy: 1.1.13
pymysql: None
psycopg2: None
jinja2: 2.9.6
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None
</code></pre></div>
<h4 dir="auto">workarounds</h4>
<p dir="auto">using</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="ax.plot(df.index.to_pydatetime(), df)"><pre class="notranslate"><span class="pl-s1">ax</span>.<span class="pl-en">plot</span>(<span class="pl-s1">df</span>.<span class="pl-s1">index</span>.<span class="pl-en">to_pydatetime</span>(), <span class="pl-s1">df</span>)</pre></div>
<p dir="auto">instead</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="ax.plot(df)"><pre class="notranslate"><span class="pl-s1">ax</span>.<span class="pl-en">plot</span>(<span class="pl-s1">df</span>)</pre></div>
<p dir="auto">This problem does not occur prior to <code class="notranslate">0.21.0</code></p>