<p dir="auto">Using pandas 0.10.1</p>
<p dir="auto">Pandas allows creating a dataframe with two columns with the same name. (I disagree that it should be allowed, but it is allowed, so OK). However, it doesn't handle that correctly in several cases.<br>
Pandas ought to either completely disallow duplicate named columns <em>or</em> handle them everywhere. But it shouldn't handle them in some cases but not others.</p>
<p dir="auto">Problem 1: Round-trip to a CSV<br>
Dump the dataframe to a CSV and then read it back. Even though duplicate columns are supposed to be legal, Pandas won't allow that in the CSV import/export.</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="df = pandas.DataFrame([[1,2]], columns=['a','a'])
#Note df has two identically named columns now
df.to_csv('foo.csv')
df2 = pandas.DataFrame.from_csv('foo.csv')
#Note df2 does NOT have the same columns anymore. the &quot;a&quot; was renamed &quot;a.1&quot;
#At the least, you ought to get some warning that dumping to a CSV will 
#lose column names."><pre class="notranslate"><span class="pl-s1">df</span> <span class="pl-c1">=</span> <span class="pl-s1">pandas</span>.<span class="pl-v">DataFrame</span>([[<span class="pl-c1">1</span>,<span class="pl-c1">2</span>]], <span class="pl-s1">columns</span><span class="pl-c1">=</span>[<span class="pl-s">'a'</span>,<span class="pl-s">'a'</span>])
<span class="pl-c">#Note df has two identically named columns now</span>
<span class="pl-s1">df</span>.<span class="pl-en">to_csv</span>(<span class="pl-s">'foo.csv'</span>)
<span class="pl-s1">df2</span> <span class="pl-c1">=</span> <span class="pl-s1">pandas</span>.<span class="pl-v">DataFrame</span>.<span class="pl-en">from_csv</span>(<span class="pl-s">'foo.csv'</span>)
<span class="pl-c">#Note df2 does NOT have the same columns anymore. the "a" was renamed "a.1"</span>
<span class="pl-c">#At the least, you ought to get some warning that dumping to a CSV will </span>
<span class="pl-c">#lose column names.</span></pre></div>
<p dir="auto">Problem 2: Merging to a dataframe with dup columns does not work</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="df = pandas.DataFrame([[1,2,3]], columns=['a','a','b'])
df.merge(df, how='left', on='b')
#Result: Very misleading error message
# edit: reindexing error"><pre class="notranslate"><span class="pl-s1">df</span> <span class="pl-c1">=</span> <span class="pl-s1">pandas</span>.<span class="pl-v">DataFrame</span>([[<span class="pl-c1">1</span>,<span class="pl-c1">2</span>,<span class="pl-c1">3</span>]], <span class="pl-s1">columns</span><span class="pl-c1">=</span>[<span class="pl-s">'a'</span>,<span class="pl-s">'a'</span>,<span class="pl-s">'b'</span>])
<span class="pl-s1">df</span>.<span class="pl-en">merge</span>(<span class="pl-s1">df</span>, <span class="pl-s1">how</span><span class="pl-c1">=</span><span class="pl-s">'left'</span>, <span class="pl-s1">on</span><span class="pl-c1">=</span><span class="pl-s">'b'</span>)
<span class="pl-c">#Result: Very misleading error message</span>
<span class="pl-c"># edit: reindexing error</span></pre></div>
<p dir="auto">I'd be ok with almost any solution (disallowing duplicate named columns, giving you a warning when you do it, handling it correctly in the merge and csv read, etc). But it should be consistently one way or the other.</p>
<h4 dir="auto">Code Sample, a copy-pastable example if possible</h4>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import pandas as pd

df = pd.DataFrame({'a': ['Hello', 'World']})

df['a'] = df['a'].astype('S')
df['b'] = df['a'].astype('S')

print(df.dtypes)"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">pandas</span> <span class="pl-k">as</span> <span class="pl-s1">pd</span>

<span class="pl-s1">df</span> <span class="pl-c1">=</span> <span class="pl-s1">pd</span>.<span class="pl-v">DataFrame</span>({<span class="pl-s">'a'</span>: [<span class="pl-s">'Hello'</span>, <span class="pl-s">'World'</span>]})

<span class="pl-s1">df</span>[<span class="pl-s">'a'</span>] <span class="pl-c1">=</span> <span class="pl-s1">df</span>[<span class="pl-s">'a'</span>].<span class="pl-en">astype</span>(<span class="pl-s">'S'</span>)
<span class="pl-s1">df</span>[<span class="pl-s">'b'</span>] <span class="pl-c1">=</span> <span class="pl-s1">df</span>[<span class="pl-s">'a'</span>].<span class="pl-en">astype</span>(<span class="pl-s">'S'</span>)

<span class="pl-en">print</span>(<span class="pl-s1">df</span>.<span class="pl-s1">dtypes</span>)</pre></div>
<p dir="auto">Output is:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="a    object
b       |S5
dtype: object"><pre class="notranslate"><code class="notranslate">a    object
b       |S5
dtype: object
</code></pre></div>
<h4 dir="auto">Problem description</h4>
<p dir="auto">The new column has the correct dtype of numpy byte string, the old column no contains<br>
the new values but still has dtype <code class="notranslate">object</code>.</p>
<h4 dir="auto">Expected Output</h4>
<p dir="auto">Both columns should have dtype 'S'</p>
<h4 dir="auto">Output of <code class="notranslate">pd.show_versions()</code></h4>
<details>
import pandas as pd
<p dir="auto">df = pd.DataFrame({'a': ['Hello', 'World']})</p>
<p dir="auto">df['a'] = df['a'].astype('S')<br>
df['b'] = df['a'].astype('S')</p>
<p dir="auto">print(df.dtypes)</p>
</details>