Stack Exchange
how does DataFrameGroupBy.apply handle large dataframes with duplicate index in pandas?
Suppose that we have a large dataframe with duplicate index,
# IPython
In [1]: import pandas as pd
In [2]: from numpy.random import randint
In [3]: df = pd.DataFrame({'a': randint(1, 10, 10000)}, index=randint(1, 10, 10000))

and we want to group by column 'a' and do some operations using apply, such as (just do nothing in apply)
In [4]: df.groupby('a').apply(lambda x: x)

which will take a long time:
In [5]: %timeit df.groupby('a').apply(lambda x: x)
19.9 s ± 322 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

If the size goes bigger, it becames unbearable. However, if we reset_index first, it goes fast.
In [6]: %timeit df.reset_index(drop=True).groupby('a').apply(lambda x: x)
2.24 ms ± 60.3 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

Then my question is how apply handle duplicate index, and why it's so slow.
Thanks for any help.

according the document.GroupBy.apply is flexible but slow.so what you intend to do.maybe we can solve it by another way.

@nay, yes. But it becomes especially slow in the case I proposed above. So I think there may be something special when it handles df with duplicate index.
