<p dir="auto">This looks like <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1667354" data-permission-text="Title is private" data-url="https://github.com/pandas-dev/pandas/issues/147" data-hovercard-type="issue" data-hovercard-url="/pandas-dev/pandas/issues/147/hovercard" href="https://github.com/pandas-dev/pandas/issues/147">#147</a>, but I don't have permission to re-open that.</p>
<p dir="auto">Consider the following situation:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="In [3]: dt
Out[3]: 
   foo  bar  baz
0  1    2    3  
1  4    5    6  
2  1    2    4  

In [4]: pandas.pivot_table(dt, values=&quot;baz&quot;, rows=[&quot;foo&quot;], cols=[&quot;bar&quot;])
Out[4]: 
  bar  2    5  
foo            
1      3.5  NaN
4      NaN  6  "><pre class="notranslate"><code class="notranslate">In [3]: dt
Out[3]: 
   foo  bar  baz
0  1    2    3  
1  4    5    6  
2  1    2    4  

In [4]: pandas.pivot_table(dt, values="baz", rows=["foo"], cols=["bar"])
Out[4]: 
  bar  2    5  
foo            
1      3.5  NaN
4      NaN  6  
</code></pre></div>
<p dir="auto">Notice the column for <code class="notranslate">foo == 1 &amp;&amp; bar == 2</code> shows a value of <code class="notranslate">3.5</code>. Either this should be in the resulting table twice, or the command should fail with an error.</p>
<p dir="auto">The dataset named 'changes' was obtained from a merge by RID:</p>
<p dir="auto">changes.head()</p>
<table role="table">
<thead>
<tr>
<th></th>
<th>DX_m12</th>
<th>RID</th>
<th>DX_m12</th>
</tr>
</thead>
<tbody>
<tr>
<td>0</td>
<td>1</td>
<td>3</td>
<td>1</td>
</tr>
<tr>
<td>1</td>
<td>0</td>
<td>4</td>
<td>0</td>
</tr>
<tr>
<td>2</td>
<td>0</td>
<td>6</td>
<td>0</td>
</tr>
<tr>
<td>3</td>
<td>0</td>
<td>8</td>
<td>0</td>
</tr>
<tr>
<td>4</td>
<td>1</td>
<td>10</td>
<td>1</td>
</tr>
</tbody>
</table>
<p dir="auto">I've used the two following commands with the intention of identifying lines in which DX_bl and DX_m12 are different:</p>
<p dir="auto">print(changes[~changes.duplicated(subset = ['DX_bl','DX_m12'], keep=False)])</p>
<table role="table">
<thead>
<tr>
<th></th>
<th>DX_bl</th>
<th>RID</th>
<th>DX_m12</th>
</tr>
</thead>
<tbody>
<tr>
<td>64</td>
<td>64</td>
<td>1</td>
<td>167</td>
</tr>
</tbody>
</table>
<p dir="auto">print(changes.drop_duplicates(subset = ['DX_bl','DX_m12']))</p>
<table role="table">
<thead>
<tr>
<th></th>
<th>DX_bl</th>
<th>RID</th>
<th>DX_m12</th>
</tr>
</thead>
<tbody>
<tr>
<td>0</td>
<td>0</td>
<td>1</td>
<td>3</td>
</tr>
<tr>
<td>1</td>
<td>1</td>
<td>0</td>
<td>4</td>
</tr>
<tr>
<td>10</td>
<td>10</td>
<td>0</td>
<td>30</td>
</tr>
<tr>
<td>64</td>
<td>64</td>
<td>1</td>
<td>167</td>
</tr>
</tbody>
</table>
<p dir="auto">If keep=False, it should return lines 10 (RID 30) and 64 (RID 64). But, as we can see, it loses the information of line 10 (RID 30).</p>
<p dir="auto">On the other hand, if 'keep' is let in default option (keep = 'first') it wrongly returns lines 0 (RID 3) and 1 (RID 4).</p>
<p dir="auto">Is there a bug in pandas' duplicated/drop_duplicates??</p>