<p dir="auto">This code all "works," but in a surprising way:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="import pandas as pd
t0 = 1234567890123400000
df1 = pd.DataFrame(index=pd.DatetimeIndex([t0, t0 + 1000, t0 + 2000, t0 + 3000]))
df2 = pd.DataFrame(index=range(4))
df1.loc[:2, 'a'] = np.arange(2)
df2.loc[:2, 'a'] = np.arange(3)"><pre class="notranslate"><code class="notranslate">import pandas as pd
t0 = 1234567890123400000
df1 = pd.DataFrame(index=pd.DatetimeIndex([t0, t0 + 1000, t0 + 2000, t0 + 3000]))
df2 = pd.DataFrame(index=range(4))
df1.loc[:2, 'a'] = np.arange(2)
df2.loc[:2, 'a'] = np.arange(3)
</code></pre></div>
<p dir="auto">We create <code class="notranslate">df1</code> with a DatetimeIndex and <code class="notranslate">df2</code> with an integer index.  We then create a new column <code class="notranslate">a</code> in each, using <code class="notranslate">.loc[]</code> with an integer slice.  With <code class="notranslate">df1</code> we get the intuitive, normal Python slice behavior where <code class="notranslate">[:2]</code> means "the first 2 elements", whereas with <code class="notranslate">df2</code> we get the bizarre—but <a href="http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.loc.html" rel="nofollow">documented</a>—<code class="notranslate">DataFrame.loc</code> slice behavior where <code class="notranslate">[:2]</code> means "elements up to index 2, inclusive."</p>
<p dir="auto">I don't see why the type of index the DataFrame has should affect the semantics of slicing with <code class="notranslate">.loc[]</code>.  I happen to think the exclusive-end behavior is correct in all cases, though apparently Pandas has decided (or at least documented) that <code class="notranslate">.loc[]</code> slicing is inclusive (in which case the DatetimeIndex case looks like a bug).</p>
<p dir="auto">Also note that trying to "read" <code class="notranslate">df1.loc[:2, 'a']</code> (e.g. to print it) fails, saying:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="TypeError: cannot do slice indexing on &lt;class 'pandas.tseries.index.DatetimeIndex'&gt;
    with these indexers [2] of &lt;class 'int'&gt;"><pre class="notranslate"><code class="notranslate">TypeError: cannot do slice indexing on &lt;class 'pandas.tseries.index.DatetimeIndex'&gt;
    with these indexers [2] of &lt;class 'int'&gt;
</code></pre></div>
<p dir="auto">It's sort of strange that you can assign to this slice but not read from it.</p>
<p dir="auto">I'm using Pandas 0.18.1.</p>
<h4 dir="auto">Code Sample, a copy-pastable example if possible</h4>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import glob
import pandas as pd

files = glob.glob('*.tab')

dfs = [pd.read_csv(fp, sep='\t',index_col=[0], header=0) for fp in files]

df = pd.merge(dfs[0], dfs[1],how='outer', left_index=True, right_index=True).sort_index()

df1 = pd.merge(dfs[2], df, how='outer', left_index=True, right_index=True).sort_index()

df2 = df1.fillna('-')

df3 = df2.drop_duplicates(keep='first') #Not working properly

print (df3)

df3.to_csv('/home/jsena/Documents/atten_sites_1.csv', sep='\t')"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">glob</span>
<span class="pl-k">import</span> <span class="pl-s1">pandas</span> <span class="pl-k">as</span> <span class="pl-s1">pd</span>

<span class="pl-s1">files</span> <span class="pl-c1">=</span> <span class="pl-s1">glob</span>.<span class="pl-en">glob</span>(<span class="pl-s">'*.tab'</span>)

<span class="pl-s1">dfs</span> <span class="pl-c1">=</span> [<span class="pl-s1">pd</span>.<span class="pl-en">read_csv</span>(<span class="pl-s1">fp</span>, <span class="pl-s1">sep</span><span class="pl-c1">=</span><span class="pl-s">'<span class="pl-cce">\t</span>'</span>,<span class="pl-s1">index_col</span><span class="pl-c1">=</span>[<span class="pl-c1">0</span>], <span class="pl-s1">header</span><span class="pl-c1">=</span><span class="pl-c1">0</span>) <span class="pl-k">for</span> <span class="pl-s1">fp</span> <span class="pl-c1">in</span> <span class="pl-s1">files</span>]

<span class="pl-s1">df</span> <span class="pl-c1">=</span> <span class="pl-s1">pd</span>.<span class="pl-en">merge</span>(<span class="pl-s1">dfs</span>[<span class="pl-c1">0</span>], <span class="pl-s1">dfs</span>[<span class="pl-c1">1</span>],<span class="pl-s1">how</span><span class="pl-c1">=</span><span class="pl-s">'outer'</span>, <span class="pl-s1">left_index</span><span class="pl-c1">=</span><span class="pl-c1">True</span>, <span class="pl-s1">right_index</span><span class="pl-c1">=</span><span class="pl-c1">True</span>).<span class="pl-en">sort_index</span>()

<span class="pl-s1">df1</span> <span class="pl-c1">=</span> <span class="pl-s1">pd</span>.<span class="pl-en">merge</span>(<span class="pl-s1">dfs</span>[<span class="pl-c1">2</span>], <span class="pl-s1">df</span>, <span class="pl-s1">how</span><span class="pl-c1">=</span><span class="pl-s">'outer'</span>, <span class="pl-s1">left_index</span><span class="pl-c1">=</span><span class="pl-c1">True</span>, <span class="pl-s1">right_index</span><span class="pl-c1">=</span><span class="pl-c1">True</span>).<span class="pl-en">sort_index</span>()

<span class="pl-s1">df2</span> <span class="pl-c1">=</span> <span class="pl-s1">df1</span>.<span class="pl-en">fillna</span>(<span class="pl-s">'-'</span>)

<span class="pl-s1">df3</span> <span class="pl-c1">=</span> <span class="pl-s1">df2</span>.<span class="pl-en">drop_duplicates</span>(<span class="pl-s1">keep</span><span class="pl-c1">=</span><span class="pl-s">'first'</span>) <span class="pl-c">#Not working properly</span>

<span class="pl-en">print</span> (<span class="pl-s1">df3</span>)

<span class="pl-s1">df3</span>.<span class="pl-en">to_csv</span>(<span class="pl-s">'/home/jsena/Documents/atten_sites_1.csv'</span>, <span class="pl-s1">sep</span><span class="pl-c1">=</span><span class="pl-s">'<span class="pl-cce">\t</span>'</span>)</pre></div>
<h4 dir="auto">Problem description</h4>
<p dir="auto">I used the code above to combine and merage several tab separated files. I then attempted to remove duplicate rows with drop_duplicates. When I use drop_duplicates, unique rows are being removed. However, If I use command-line tools I can get the correct output.</p>
<p dir="auto">Thank you very much for your help!</p>
<p dir="auto"><strong>Input Files:</strong><br>
<a href="https://github.com/pandas-dev/pandas/files/1741608/5270.txt">5270.txt</a><br>
<a href="https://github.com/pandas-dev/pandas/files/1741609/2579.txt">2579.txt</a><br>
<a href="https://github.com/pandas-dev/pandas/files/1741610/57.txt">57.txt</a><br>
<strong>note:</strong>  I had to change the extension from .tab to .txt to attach the files.</p>
<p dir="auto"><strong>Note</strong>: Many problems can be resolved by simply upgrading <code class="notranslate">pandas</code> to the latest version. Before submitting, please check if that solution works for you. If possible, you may want to check if <code class="notranslate">master</code> addresses this issue, but that is not necessary.</p>
<p dir="auto"><em>Upgrading Pandas did not solve my issue. I get the same results with conda and pandas</em><br>
<strong>Pandas version used with conda</strong><br>
pd.<strong>version</strong><br>
'0.22.0'</p>
<p dir="auto"><strong>Expected Output</strong><br>
<a href="https://github.com/pandas-dev/pandas/files/1749880/expected_output.txt">expected_output.txt</a></p>
<p dir="auto">read	 56	 2578	5269<br>
100005/ccs#CACTAGTCTGTTATTC	              T	A	T<br>
100009/ccs#CCACAATGGCGGCACT	              T	A	T<br>
100013/ccs#CACCTCCACCGTTAAT	              T	A	T<br>
00017/ccs#CCTTCACAGTATAACA	                      T	-	T<br>
100021/ccs#ACCTTGAATCGTCGAC	              T	A	T<br>
100023/ccs#CCGACTGTCATATGGT	              T	A	T<br>
100027/ccs#CCGCATTCCGAAATTA	              T	A	T<br>
10004/ccs#CTCTTGCACAAAATCG	              T	A	T</p>
<p dir="auto"><strong>drop_duplicates output</strong><br>
<a href="https://github.com/pandas-dev/pandas/files/1741647/pandas_output.txt">pandas_output.txt</a></p>
<p dir="auto">read	56	2578	5269<br>
100005/ccs#CACTAGTCTGTTATTC	T	A	T<br>
100017/ccs#CCTTCACAGTATAACA	T	-	T<br>
100075/ccs#CTCTGTAAGTGTCCTT	-	A	T<br>
10008/ccs#CCAGCTTCGGACCAGC	T	A	-<br>
10052/ccs#CCCATCCGGTCGAAAC	-	-	T<br>
100575/ccs#CGGTACTTGTGTTTTC	-	A	-<br>
100734/ccs#CCGCACTCCTGCTCAC	T	-	-<br>
101808/ccs#CATGCGATCGAAACCG	C	A	T</p>
<h4 dir="auto">Output of <code class="notranslate">pd.show_versions()</code></h4>
<p dir="auto">Python and Pandas version<br>
Python 3.6.3 |Anaconda, Inc.| (default, Oct 13 2017, 12:02:49)<br>
[GCC 7.2.0] on linux<br>
pandas.<strong>version</strong><br>
'0.20.3'</p>