title: How to remove duplicate groups in a pandas dataframe?
body: <p>I would like to remove duplicated sets in the dataframe.</p>
  
  <pre><code>import pandas as pd
  import pdb
  filename = "result_4_tiling_116.csv"
  
  chunksize = 10 ** 6
  for chunk in pd.read_csv(filename, names=['id', 'tileID', 'x', 'y', 'h', 'w'], chunksize=chunksize):
      pdb.set_trace()
  </code></pre>
  
  <p>An example of the first 31 lines of the data:</p>
  
  <pre><code>chunk.head(31)
  
        tileID  x     y  h    w
  0          0  0   0.0  1  8.0
  1          1  0   8.0  1  8.0
  2          0  0   8.0  1  8.0
  3          1  0   0.0  1  4.0
  4          2  0   4.0  1  4.0
  5          0  0   0.0  1  4.0
  6          1  0   4.0  1  4.0
  7          2  0   8.0  1  4.0
  8          3  0  12.0  1  4.0
  9          0  0   4.0  1  4.0
  10         1  0   8.0  1  4.0
  11         2  0  12.0  1  4.0
  12         3  0   0.0  1  2.0
  13         4  0   2.0  1  2.0
  14         0  0   8.0  1  4.0
  15         1  0  12.0  1  4.0
  16         2  0   0.0  1  2.0
  17         3  0   2.0  1  2.0
  18         4  0   4.0  1  2.0
  19         5  0   6.0  1  2.0
  20         0  0  12.0  1  4.0
  21         1  0   0.0  1  2.0
  22         2  0   2.0  1  2.0
  23         3  0   4.0  1  2.0
  24         4  0   6.0  1  2.0
  25         0  0   8.0  1  4.0
  26         1  0  12.0  1  4.0
  27         2  0   0.0  1  2.0
  28         3  0   2.0  1  2.0
  29         4  0   4.0  1  2.0
  30         5  0   6.0  1  2.0
  </code></pre>
  
  <p>I would like to filter out the duplicated ones. The data contains a set of groups (for each starting with tileID=0), as follows: 
  1.</p>
  
  <pre><code>0          0  0   0.0  1  8.0
  1          1  0   8.0  1  8.0
  </code></pre>
  
  <p>2.</p>
  
  <pre><code>2          0  0   8.0  1  8.0
  3          1  0   0.0  1  4.0
  4          2  0   4.0  1  4.0
  </code></pre>
  
  <p>3.</p>
  
  <pre><code>5          0  0   0.0  1  4.0
  6          1  0   4.0  1  4.0
  7          2  0   8.0  1  4.0
  8          3  0  12.0  1  4.0
  </code></pre>
  
  <p>4.</p>
  
  <pre><code>9          0  0   4.0  1  4.0
  10         1  0   8.0  1  4.0
  11         2  0  12.0  1  4.0
  12         3  0   0.0  1  2.0
  13         4  0   2.0  1  2.0
  </code></pre>
  
  <p>5.</p>
  
  <pre><code>14         0  0   8.0  1  4.0
  15         1  0  12.0  1  4.0
  16         2  0   0.0  1  2.0
  17         3  0   2.0  1  2.0
  18         4  0   4.0  1  2.0
  19         5  0   6.0  1  2.0
  </code></pre>
  
  <p>6.</p>
  
  <pre><code>20         0  0  12.0  1  4.0
  21         1  0   0.0  1  2.0
  22         2  0   2.0  1  2.0
  23         3  0   4.0  1  2.0
  24         4  0   6.0  1  2.0
  </code></pre>
  
  <p>7.</p>
  
  <pre><code>25         0  0   8.0  1  4.0
  26         1  0  12.0  1  4.0
  27         2  0   0.0  1  2.0
  28         3  0   2.0  1  2.0
  29         4  0   4.0  1  2.0
  30         5  0   6.0  1  2.0
  </code></pre>
  
  <p>In this example, 5 and 7 are duplicated data. I try to use <code>drop_duplicates</code>, but no success yet. </p>