title: Determining duplicate values in an array
body: <p>Suppose I have an array</p>
  
  <pre><code>a = np.array([1, 2, 1, 3, 3, 3, 0])
  </code></pre>
  
  <p>How can I (efficiently, Pythonically) find which elements of <code>a</code> are duplicates (i.e., non-unique values)?  In this case the result would be <code>array([1, 3, 3])</code> or possibly <code>array([1, 3])</code> if efficient.</p>
  
  <p>I've come up with a few methods that appear to work:</p>
  
  <h3>Masking</h3>
  
  <pre><code>m = np.zeros_like(a, dtype=bool)
  m[np.unique(a, return_index=True)[1]] = True
  a[~m]
  </code></pre>
  
  <h3>Set operations</h3>
  
  <pre><code>a[~np.in1d(np.arange(len(a)), np.unique(a, return_index=True)[1], assume_unique=True)]
  </code></pre>
  
  <p>This one is cute but probably illegal (as <code>a</code> isn't actually unique):</p>
  
  <pre><code>np.setxor1d(a, np.unique(a), assume_unique=True)
  </code></pre>
  
  <h3>Histograms</h3>
  
  <pre><code>u, i = np.unique(a, return_inverse=True)
  u[np.bincount(i) &gt; 1]
  </code></pre>
  
  <h3>Sorting</h3>
  
  <pre><code>s = np.sort(a, axis=None)
  s[:-1][s[1:] == s[:-1]]
  </code></pre>
  
  <h3>Pandas</h3>
  
  <pre><code>s = pd.Series(a)
  s[s.duplicated()]
  </code></pre>
  
  <p>Is there anything I've missed?  I'm not necessarily looking for a numpy-only solution, but it has to work with numpy data types and be efficient on medium-sized data sets (up to 10 million in size).</p>
  
  <hr/>
  
  <h2>Conclusions</h2>
  
  <p>Testing with a 10 million size data set (on a 2.8GHz Xeon):</p>
  
  <pre><code>a = np.random.randint(10**7, size=10**7)
  </code></pre>
  
  <p>The fastest is sorting, at 1.1s.  The dubious <code>xor1d</code> is second at 2.6s, followed by masking and Pandas <code>Series.duplicated</code> at 3.1s, <code>bincount</code> at 5.6s, and <code>in1d</code> and senderle's <code>setdiff1d</code> both at 7.3s.  Steven's <code>Counter</code> is only a little slower, at 10.5s; trailing behind are Burhan's <code>Counter.most_common</code> at 110s and DSM's <code>Counter</code> subtraction at 360s.</p>
  
  <p>I'm going to use sorting for performance, but I'm accepting Steven's answer because the performance is acceptable and it <em>feels</em> clearer and more Pythonic.</p>
  
  <p>Edit: discovered the Pandas solution.  If Pandas is available it's clear and performs well.</p>