<h4 dir="auto">Code Sample, a copy-pastable example if possible</h4>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import pandas as pd

# create our dataframe
m = 5
temp = pd.DataFrame({
        'a': ['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j'] * m,
        'b': ['t', 'w', 'x', 'y', 'z'] * 2 * m,
        'c': [letter 
              for each in ['m', 'n', 'u',  'p', 'o']
              for letter in [each] * 2 * m],
        'd': [letter 
              for each in ['aa', 'bb', 'cc', 'dd', 'ee',
                           'ff', 'gg', 'hh','ii', 'jj']
              for letter in [each] * m],

    })

# change them all to categorical variables
for c in temp.columns:
    temp[c] = temp[c].astype('category')

# get the dimensions before we do anything
print(temp.shape)

# drop duplicates to make sure this is unqiue
# it should be unique
id_df = temp.drop_duplicates()
print(id_df.shape)

# join a row-wise unique dataset to itself on all variables
# when they're categorical variables it duplicates rows
# when they're strings things behave as they're suppposed to
temp1 = pd.merge(temp, id_df, on = list(temp.columns))
print(temp1.shape)
"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">pandas</span> <span class="pl-k">as</span> <span class="pl-s1">pd</span>

<span class="pl-c"># create our dataframe</span>
<span class="pl-s1">m</span> <span class="pl-c1">=</span> <span class="pl-c1">5</span>
<span class="pl-s1">temp</span> <span class="pl-c1">=</span> <span class="pl-s1">pd</span>.<span class="pl-v">DataFrame</span>({
        <span class="pl-s">'a'</span>: [<span class="pl-s">'a'</span>, <span class="pl-s">'b'</span>, <span class="pl-s">'c'</span>, <span class="pl-s">'d'</span>, <span class="pl-s">'e'</span>, <span class="pl-s">'f'</span>, <span class="pl-s">'g'</span>, <span class="pl-s">'h'</span>, <span class="pl-s">'i'</span>, <span class="pl-s">'j'</span>] <span class="pl-c1">*</span> <span class="pl-s1">m</span>,
        <span class="pl-s">'b'</span>: [<span class="pl-s">'t'</span>, <span class="pl-s">'w'</span>, <span class="pl-s">'x'</span>, <span class="pl-s">'y'</span>, <span class="pl-s">'z'</span>] <span class="pl-c1">*</span> <span class="pl-c1">2</span> <span class="pl-c1">*</span> <span class="pl-s1">m</span>,
        <span class="pl-s">'c'</span>: [<span class="pl-s1">letter</span> 
              <span class="pl-k">for</span> <span class="pl-s1">each</span> <span class="pl-c1">in</span> [<span class="pl-s">'m'</span>, <span class="pl-s">'n'</span>, <span class="pl-s">'u'</span>,  <span class="pl-s">'p'</span>, <span class="pl-s">'o'</span>]
              <span class="pl-k">for</span> <span class="pl-s1">letter</span> <span class="pl-c1">in</span> [<span class="pl-s1">each</span>] <span class="pl-c1">*</span> <span class="pl-c1">2</span> <span class="pl-c1">*</span> <span class="pl-s1">m</span>],
        <span class="pl-s">'d'</span>: [<span class="pl-s1">letter</span> 
              <span class="pl-k">for</span> <span class="pl-s1">each</span> <span class="pl-c1">in</span> [<span class="pl-s">'aa'</span>, <span class="pl-s">'bb'</span>, <span class="pl-s">'cc'</span>, <span class="pl-s">'dd'</span>, <span class="pl-s">'ee'</span>,
                           <span class="pl-s">'ff'</span>, <span class="pl-s">'gg'</span>, <span class="pl-s">'hh'</span>,<span class="pl-s">'ii'</span>, <span class="pl-s">'jj'</span>]
              <span class="pl-k">for</span> <span class="pl-s1">letter</span> <span class="pl-c1">in</span> [<span class="pl-s1">each</span>] <span class="pl-c1">*</span> <span class="pl-s1">m</span>],

    })

<span class="pl-c"># change them all to categorical variables</span>
<span class="pl-k">for</span> <span class="pl-s1">c</span> <span class="pl-c1">in</span> <span class="pl-s1">temp</span>.<span class="pl-s1">columns</span>:
    <span class="pl-s1">temp</span>[<span class="pl-s1">c</span>] <span class="pl-c1">=</span> <span class="pl-s1">temp</span>[<span class="pl-s1">c</span>].<span class="pl-en">astype</span>(<span class="pl-s">'category'</span>)

<span class="pl-c"># get the dimensions before we do anything</span>
<span class="pl-en">print</span>(<span class="pl-s1">temp</span>.<span class="pl-s1">shape</span>)

<span class="pl-c"># drop duplicates to make sure this is unqiue</span>
<span class="pl-c"># it should be unique</span>
<span class="pl-s1">id_df</span> <span class="pl-c1">=</span> <span class="pl-s1">temp</span>.<span class="pl-en">drop_duplicates</span>()
<span class="pl-en">print</span>(<span class="pl-s1">id_df</span>.<span class="pl-s1">shape</span>)

<span class="pl-c"># join a row-wise unique dataset to itself on all variables</span>
<span class="pl-c"># when they're categorical variables it duplicates rows</span>
<span class="pl-c"># when they're strings things behave as they're suppposed to</span>
<span class="pl-s1">temp1</span> <span class="pl-c1">=</span> <span class="pl-s1">pd</span>.<span class="pl-en">merge</span>(<span class="pl-s1">temp</span>, <span class="pl-s1">id_df</span>, <span class="pl-s1">on</span> <span class="pl-c1">=</span> <span class="pl-en">list</span>(<span class="pl-s1">temp</span>.<span class="pl-s1">columns</span>))
<span class="pl-en">print</span>(<span class="pl-s1">temp1</span>.<span class="pl-s1">shape</span>)</pre></div>
<h4 dir="auto">Problem description</h4>
<p dir="auto">Using merge on Categorical dtypes doesn't appear to be checking equality correctly.  Merging a unique dataframe to itself on 4 Categorical columns appears to duplicate rows.  The above code example is simpler than what I experienced the issue on but the behavior is there.</p>
<p dir="auto">The dataframe as it is created is a 50 row by 4 column dataframe of strings.  Casting the strings to Categoricals to save on RAM appears to work well.  Running the drop_duplicates method and checking the dimensions shows that each row is unique.  Then simply merging the dataframes together results in a 54 row by 4 column dataframe.</p>
<p dir="auto">My guess is that there is something about the way the values are assigned that underlie the labels differs and that the underlying values may be equal when the labels aren't.  It appears to be a fairly specific case, as commenting any of those columns out results in what I'd expect in terms of output.</p>
<h4 dir="auto">Expected Output</h4>
<p dir="auto">Running the same code and steps as illustrated in the prior paragraph without casting the columns to a Categorical dtype results in what I would expect: a 50 row by 4 column dataframe.</p>
<h4 dir="auto">Output of <code class="notranslate">pd.show_versions()</code></h4>
<details>
INSTALLED VERSIONS
------------------
commit: None
python: 3.5.1.final.0
python-bits: 64
OS: Windows
OS-release: 10
machine: AMD64
processor: Intel64 Family 6 Model 94 Stepping 3, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None
LOCALE: None.None
<p dir="auto">pandas: 0.20.2<br>
pytest: 2.8.5<br>
pip: 9.0.1<br>
setuptools: 20.3<br>
Cython: 0.23.4<br>
numpy: 1.13.0<br>
scipy: 0.17.0<br>
xarray: None<br>
IPython: 4.1.2<br>
sphinx: 1.3.1<br>
patsy: 0.4.0<br>
dateutil: 2.6.0<br>
pytz: 2017.2<br>
blosc: None<br>
bottleneck: 1.0.0<br>
tables: 3.2.2<br>
numexpr: 2.5<br>
feather: None<br>
matplotlib: 1.5.1<br>
openpyxl: 2.3.2<br>
xlrd: 0.9.4<br>
xlwt: 1.0.0<br>
xlsxwriter: 0.8.4<br>
lxml: 3.6.0<br>
bs4: 4.4.1<br>
html5lib: None<br>
sqlalchemy: 1.0.12<br>
pymysql: None<br>
psycopg2: None<br>
jinja2: 2.8<br>
s3fs: None<br>
pandas_gbq: None<br>
pandas_datareader: None</p>
</details>
<h4 dir="auto">Code Sample</h4>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="s1 = pd.Series([1,1,2,2,3,3], name='s')
s2 = pd.Series([1,1,1,2,2,2], name='s')

pd.crosstab(s1, s2)"><pre class="notranslate"><span class="pl-s1">s1</span> <span class="pl-c1">=</span> <span class="pl-s1">pd</span>.<span class="pl-v">Series</span>([<span class="pl-c1">1</span>,<span class="pl-c1">1</span>,<span class="pl-c1">2</span>,<span class="pl-c1">2</span>,<span class="pl-c1">3</span>,<span class="pl-c1">3</span>], <span class="pl-s1">name</span><span class="pl-c1">=</span><span class="pl-s">'s'</span>)
<span class="pl-s1">s2</span> <span class="pl-c1">=</span> <span class="pl-s1">pd</span>.<span class="pl-v">Series</span>([<span class="pl-c1">1</span>,<span class="pl-c1">1</span>,<span class="pl-c1">1</span>,<span class="pl-c1">2</span>,<span class="pl-c1">2</span>,<span class="pl-c1">2</span>], <span class="pl-s1">name</span><span class="pl-c1">=</span><span class="pl-s">'s'</span>)

<span class="pl-s1">pd</span>.<span class="pl-en">crosstab</span>(<span class="pl-s1">s1</span>, <span class="pl-s1">s2</span>)</pre></div>
<p dir="auto"><strong>Return:</strong></p>
<p dir="auto">Short version:</p>
<blockquote>
<p dir="auto">ValueError: Duplicated level name: "s", assigned to level 1, is already used for level 0.</p>
</blockquote>
<p dir="auto">Long version:</p>
<details>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
&lt;ipython-input-66-b5484aa990df&gt; in &lt;module&gt;()
      2 s2 = pd.Series([1,1,1,2,2,2], name='s')
      3 
----&gt; 4 pd.crosstab(s1, s2)

C:\A_PROGRAMS\anaconda3\lib\site-packages\pandas\core\reshape\pivot.py in crosstab(index, columns, values, rownames, colnames, aggfunc, margins, margins_name, dropna, normalize)
    490     table = df.pivot_table('__dummy__', index=rownames, columns=colnames,
    491                            margins=margins, margins_name=margins_name,
--&gt; 492                            dropna=dropna, **kwargs)
    493 
    494     # Post-process

C:\A_PROGRAMS\anaconda3\lib\site-packages\pandas\core\frame.py in pivot_table(self, values, index, columns, aggfunc, fill_value, margins, dropna, margins_name)
   5301                            aggfunc=aggfunc, fill_value=fill_value,
   5302                            margins=margins, dropna=dropna,
-&gt; 5303                            margins_name=margins_name)
   5304 
   5305     def stack(self, level=-1, dropna=True):

C:\A_PROGRAMS\anaconda3\lib\site-packages\pandas\core\reshape\pivot.py in pivot_table(data, values, index, columns, aggfunc, fill_value, margins, dropna, margins_name)
     85     # if we have a categorical
     86     grouped = data.groupby(keys, observed=False)
---&gt; 87     agged = grouped.agg(aggfunc)
     88     if dropna and isinstance(agged, ABCDataFrame) and len(agged.columns):
     89         agged = agged.dropna(how='all')

C:\A_PROGRAMS\anaconda3\lib\site-packages\pandas\core\groupby\groupby.py in aggregate(self, arg, *args, **kwargs)
   4656         axis=''))
   4657     def aggregate(self, arg, *args, **kwargs):
-&gt; 4658         return super(DataFrameGroupBy, self).aggregate(arg, *args, **kwargs)
   4659 
   4660     agg = aggregate

C:\A_PROGRAMS\anaconda3\lib\site-packages\pandas\core\groupby\groupby.py in aggregate(self, arg, *args, **kwargs)
   4095             # grouper specific aggregations
   4096             if self.grouper.nkeys &gt; 1:
-&gt; 4097                 return self._python_agg_general(arg, *args, **kwargs)
   4098             else:
   4099 

C:\A_PROGRAMS\anaconda3\lib\site-packages\pandas\core\groupby\groupby.py in _python_agg_general(self, func, *args, **kwargs)
   1086                 output[name] = self._try_cast(values[mask], result)
   1087 
-&gt; 1088         return self._wrap_aggregated_output(output)
   1089 
   1090     def _wrap_applied_output(self, *args, **kwargs):

C:\A_PROGRAMS\anaconda3\lib\site-packages\pandas\core\groupby\groupby.py in _wrap_aggregated_output(self, output, names)
   4728             result = result._consolidate()
   4729         else:
-&gt; 4730             index = self.grouper.result_index
   4731             result = DataFrame(output, index=index, columns=output_keys)
   4732 

pandas/_libs/properties.pyx in pandas._libs.properties.CachedProperty.__get__()

C:\A_PROGRAMS\anaconda3\lib\site-packages\pandas\core\groupby\groupby.py in result_index(self)
   2379                             labels=labels,
   2380                             verify_integrity=False,
-&gt; 2381                             names=self.names)
   2382         return result
   2383 

C:\A_PROGRAMS\anaconda3\lib\site-packages\pandas\core\indexes\multi.py in __new__(cls, levels, labels, sortorder, names, dtype, copy, name, verify_integrity, _set_identity)
    230         if names is not None:
    231             # handles name validation
--&gt; 232             result._set_names(names)
    233 
    234         if sortorder is not None:

C:\A_PROGRAMS\anaconda3\lib\site-packages\pandas\core\indexes\multi.py in _set_names(self, names, level, validate)
    693                         'Duplicated level name: &quot;{}&quot;, assigned to '
    694                         'level {}, is already used for level '
--&gt; 695                         '{}.'.format(name, l, used[name]))
    696 
    697             self.levels[l].rename(name, inplace=True)

ValueError: Duplicated level name: &quot;s&quot;, assigned to level 1, is already used for level 0."><pre class="notranslate"><code class="notranslate">---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
&lt;ipython-input-66-b5484aa990df&gt; in &lt;module&gt;()
      2 s2 = pd.Series([1,1,1,2,2,2], name='s')
      3 
----&gt; 4 pd.crosstab(s1, s2)

C:\A_PROGRAMS\anaconda3\lib\site-packages\pandas\core\reshape\pivot.py in crosstab(index, columns, values, rownames, colnames, aggfunc, margins, margins_name, dropna, normalize)
    490     table = df.pivot_table('__dummy__', index=rownames, columns=colnames,
    491                            margins=margins, margins_name=margins_name,
--&gt; 492                            dropna=dropna, **kwargs)
    493 
    494     # Post-process

C:\A_PROGRAMS\anaconda3\lib\site-packages\pandas\core\frame.py in pivot_table(self, values, index, columns, aggfunc, fill_value, margins, dropna, margins_name)
   5301                            aggfunc=aggfunc, fill_value=fill_value,
   5302                            margins=margins, dropna=dropna,
-&gt; 5303                            margins_name=margins_name)
   5304 
   5305     def stack(self, level=-1, dropna=True):

C:\A_PROGRAMS\anaconda3\lib\site-packages\pandas\core\reshape\pivot.py in pivot_table(data, values, index, columns, aggfunc, fill_value, margins, dropna, margins_name)
     85     # if we have a categorical
     86     grouped = data.groupby(keys, observed=False)
---&gt; 87     agged = grouped.agg(aggfunc)
     88     if dropna and isinstance(agged, ABCDataFrame) and len(agged.columns):
     89         agged = agged.dropna(how='all')

C:\A_PROGRAMS\anaconda3\lib\site-packages\pandas\core\groupby\groupby.py in aggregate(self, arg, *args, **kwargs)
   4656         axis=''))
   4657     def aggregate(self, arg, *args, **kwargs):
-&gt; 4658         return super(DataFrameGroupBy, self).aggregate(arg, *args, **kwargs)
   4659 
   4660     agg = aggregate

C:\A_PROGRAMS\anaconda3\lib\site-packages\pandas\core\groupby\groupby.py in aggregate(self, arg, *args, **kwargs)
   4095             # grouper specific aggregations
   4096             if self.grouper.nkeys &gt; 1:
-&gt; 4097                 return self._python_agg_general(arg, *args, **kwargs)
   4098             else:
   4099 

C:\A_PROGRAMS\anaconda3\lib\site-packages\pandas\core\groupby\groupby.py in _python_agg_general(self, func, *args, **kwargs)
   1086                 output[name] = self._try_cast(values[mask], result)
   1087 
-&gt; 1088         return self._wrap_aggregated_output(output)
   1089 
   1090     def _wrap_applied_output(self, *args, **kwargs):

C:\A_PROGRAMS\anaconda3\lib\site-packages\pandas\core\groupby\groupby.py in _wrap_aggregated_output(self, output, names)
   4728             result = result._consolidate()
   4729         else:
-&gt; 4730             index = self.grouper.result_index
   4731             result = DataFrame(output, index=index, columns=output_keys)
   4732 

pandas/_libs/properties.pyx in pandas._libs.properties.CachedProperty.__get__()

C:\A_PROGRAMS\anaconda3\lib\site-packages\pandas\core\groupby\groupby.py in result_index(self)
   2379                             labels=labels,
   2380                             verify_integrity=False,
-&gt; 2381                             names=self.names)
   2382         return result
   2383 

C:\A_PROGRAMS\anaconda3\lib\site-packages\pandas\core\indexes\multi.py in __new__(cls, levels, labels, sortorder, names, dtype, copy, name, verify_integrity, _set_identity)
    230         if names is not None:
    231             # handles name validation
--&gt; 232             result._set_names(names)
    233 
    234         if sortorder is not None:

C:\A_PROGRAMS\anaconda3\lib\site-packages\pandas\core\indexes\multi.py in _set_names(self, names, level, validate)
    693                         'Duplicated level name: "{}", assigned to '
    694                         'level {}, is already used for level '
--&gt; 695                         '{}.'.format(name, l, used[name]))
    696 
    697             self.levels[l].rename(name, inplace=True)

ValueError: Duplicated level name: "s", assigned to level 1, is already used for level 0.
</code></pre></div>
</details>
<h4 dir="auto">Problem description</h4>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="27303661" data-permission-text="Title is private" data-url="https://github.com/pandas-dev/pandas/issues/6319" data-hovercard-type="issue" data-hovercard-url="/pandas-dev/pandas/issues/6319/hovercard" href="https://github.com/pandas-dev/pandas/issues/6319">#6319</a>  supposedly fixed this issue, yet it still persists in my configuration.</p>
<h4 dir="auto">Output of <code class="notranslate">pd.show_versions()</code></h4>
<details>
<h2 dir="auto">INSTALLED VERSIONS</h2>
<p dir="auto">commit: None<br>
python: 3.6.5.final.0<br>
python-bits: 64<br>
OS: Windows<br>
OS-release: 7<br>
machine: AMD64<br>
processor: Intel64 Family 6 Model 94 Stepping 3, GenuineIntel<br>
byteorder: little<br>
LC_ALL: None<br>
LANG: None<br>
LOCALE: None.None</p>
<p dir="auto">pandas: 0.23.1<br>
pytest: 3.5.1<br>
pip: 10.0.1<br>
setuptools: 39.1.0<br>
Cython: 0.28.2<br>
numpy: 1.14.3<br>
scipy: 1.1.0<br>
pyarrow: 0.9.0<br>
xarray: None<br>
IPython: 6.4.0<br>
sphinx: 1.7.4<br>
patsy: 0.5.0<br>
dateutil: 2.7.3<br>
pytz: 2018.4<br>
blosc: None<br>
bottleneck: 1.2.1<br>
tables: 3.4.2<br>
numexpr: 2.6.5<br>
feather: 0.4.0<br>
matplotlib: 2.1.2<br>
openpyxl: 2.5.3<br>
xlrd: 1.1.0<br>
xlwt: 1.3.0<br>
xlsxwriter: 1.0.4<br>
lxml: 4.1.1<br>
bs4: 4.6.0<br>
html5lib: 1.0.1<br>
sqlalchemy: 1.2.7<br>
pymysql: None<br>
psycopg2: None<br>
jinja2: 2.10<br>
s3fs: None<br>
fastparquet: None<br>
pandas_gbq: None<br>
pandas_datareader: None</p>
</details>