<p dir="auto">This issue is created based on the discussion from <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="220092580" data-permission-text="Title is private" data-url="https://github.com/pandas-dev/pandas/issues/15931" data-hovercard-type="pull_request" data-hovercard-url="/pandas-dev/pandas/pull/15931/hovercard" href="https://github.com/pandas-dev/pandas/pull/15931">#15931</a> following the deprecation of relabeling dicts in <code class="notranslate">groupby.agg</code>. A lot of what is summarized below was already discussed in the previous discussion. I would recommend in particular <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="220092580" data-permission-text="Title is private" data-url="https://github.com/pandas-dev/pandas/issues/15931" data-hovercard-type="pull_request" data-hovercard-url="/pandas-dev/pandas/pull/15931/hovercard?comment_id=336139085&amp;comment_type=issue_comment" href="https://github.com/pandas-dev/pandas/pull/15931#issuecomment-336139085">#15931 (comment)</a> where the problems are also clearly stated.</p>
<p dir="auto">The motivation behind the deprecation of <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="220092580" data-permission-text="Title is private" data-url="https://github.com/pandas-dev/pandas/issues/15931" data-hovercard-type="pull_request" data-hovercard-url="/pandas-dev/pandas/pull/15931/hovercard" href="https://github.com/pandas-dev/pandas/pull/15931">#15931</a> was mostly related to bringing a consistent interface for <code class="notranslate">agg()</code> between Series and Dataframe (see also <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="189561770" data-permission-text="Title is private" data-url="https://github.com/pandas-dev/pandas/issues/14668" data-hovercard-type="pull_request" data-hovercard-url="/pandas-dev/pandas/pull/14668/hovercard" href="https://github.com/pandas-dev/pandas/pull/14668">#14668</a> for context).</p>
<p dir="auto">The relabeling functionality with a nested dict has been described by some as being too complex and/or inconsistent and thus deprecated.</p>
<p dir="auto">However, this comes at a price: the impossibility to aggregate and rename at the same time leads to very annoying issues and some backward incompatibility where no sensible workaround is available:</p>
<ul dir="auto">
<li><em>[annoying]</em> no more control over the names of the resulting columns</li>
<li><em>[annoying]</em> you need to find a way to rename the MultiIndex <em>after</em> performing the aggregation, requiring to keep track of the order of columns at two places in the code.... not practical at all and sometimes downright impossible (cases below).</li>
<li><g-emoji class="g-emoji" alias="warning" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/26a0.png">⚠️</g-emoji>  <em><strong>[breaking]</strong></em> cannot apply more than one callable with the same internal name on the same input column. This results in two sub-cases:
<ul dir="auto">
<li><em><strong>[breaking]</strong></em> you can't apply anymore two or more lambda aggregators on the same column</li>
<li><em><strong>[breaking]</strong></em> you can't apply anymore two or more aggregators from partial functions unless you alter their hidden <code class="notranslate">__name__</code> attribute</li>
</ul>
</li>
</ul>
<h2 dir="auto">Example</h2>
<p dir="auto"><em>(please note, this is a crafted example for the purpose of demonstrating the problem in as short a code as possible, but all of the demonstrated issues here did bite me in real life since the change, and in situations not as simple as here)</em></p>
<h3 dir="auto">Input Dataframe</h3>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="mydf = pd.DataFrame(
    {
        'cat': ['A', 'A', 'A', 'B', 'B', 'C'],
        'energy': [1.8, 1.95, 2.04, 1.25, 1.6, 1.01],
        'distance': [1.2, 1.5, 1.74, 0.82, 1.01, 0.6]
    },
    index=range(6)
)"><pre class="notranslate"><span class="pl-s1">mydf</span> <span class="pl-c1">=</span> <span class="pl-s1">pd</span>.<span class="pl-v">DataFrame</span>(
    {
        <span class="pl-s">'cat'</span>: [<span class="pl-s">'A'</span>, <span class="pl-s">'A'</span>, <span class="pl-s">'A'</span>, <span class="pl-s">'B'</span>, <span class="pl-s">'B'</span>, <span class="pl-s">'C'</span>],
        <span class="pl-s">'energy'</span>: [<span class="pl-c1">1.8</span>, <span class="pl-c1">1.95</span>, <span class="pl-c1">2.04</span>, <span class="pl-c1">1.25</span>, <span class="pl-c1">1.6</span>, <span class="pl-c1">1.01</span>],
        <span class="pl-s">'distance'</span>: [<span class="pl-c1">1.2</span>, <span class="pl-c1">1.5</span>, <span class="pl-c1">1.74</span>, <span class="pl-c1">0.82</span>, <span class="pl-c1">1.01</span>, <span class="pl-c1">0.6</span>]
    },
    <span class="pl-s1">index</span><span class="pl-c1">=</span><span class="pl-en">range</span>(<span class="pl-c1">6</span>)
)</pre></div>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="  cat  distance  energy
0   A      1.20    1.80
1   A      1.50    1.95
2   A      1.74    2.04
3   B      0.82    1.25
4   B      1.01    1.60
5   C      0.60    1.01"><pre class="notranslate"><code class="notranslate">  cat  distance  energy
0   A      1.20    1.80
1   A      1.50    1.95
2   A      1.74    2.04
3   B      0.82    1.25
4   B      1.01    1.60
5   C      0.60    1.01
</code></pre></div>
<h3 dir="auto">Before:</h3>
<p dir="auto">easy to write and read, and works as expected</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import numpy as np
import statsmodels.robust as smrb
from functools import partial
 
# median absolute deviation as a partial function
# in order to demonstrate the issue with partial functions as aggregators
mad_c1 = partial(smrb.mad, c=1)

# renaming and specifying the aggregators at the same time
# note that I want to choose the resulting column names myself
# for example &quot;total_xxxx&quot; instead of just &quot;sum&quot;
mydf_agg = mydf.groupby('cat').agg({
    'energy': {
        'total_energy': 'sum',
        'energy_p98': lambda x: np.percentile(x, 98),  # lambda
        'energy_p17': lambda x: np.percentile(x, 17),  # lambda
    },
    'distance': {
        'total_distance': 'sum',
        'average_distance': 'mean',
        'distance_mad': smrb.mad,   # original function
        'distance_mad_c1': mad_c1,  # partial function wrapping the original function
    },
})"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">numpy</span> <span class="pl-k">as</span> <span class="pl-s1">np</span>
<span class="pl-k">import</span> <span class="pl-s1">statsmodels</span>.<span class="pl-s1">robust</span> <span class="pl-k">as</span> <span class="pl-s1">smrb</span>
<span class="pl-k">from</span> <span class="pl-s1">functools</span> <span class="pl-k">import</span> <span class="pl-s1">partial</span>
 
<span class="pl-c"># median absolute deviation as a partial function</span>
<span class="pl-c"># in order to demonstrate the issue with partial functions as aggregators</span>
<span class="pl-s1">mad_c1</span> <span class="pl-c1">=</span> <span class="pl-en">partial</span>(<span class="pl-s1">smrb</span>.<span class="pl-s1">mad</span>, <span class="pl-s1">c</span><span class="pl-c1">=</span><span class="pl-c1">1</span>)

<span class="pl-c"># renaming and specifying the aggregators at the same time</span>
<span class="pl-c"># note that I want to choose the resulting column names myself</span>
<span class="pl-c"># for example "total_xxxx" instead of just "sum"</span>
<span class="pl-s1">mydf_agg</span> <span class="pl-c1">=</span> <span class="pl-s1">mydf</span>.<span class="pl-en">groupby</span>(<span class="pl-s">'cat'</span>).<span class="pl-en">agg</span>({
    <span class="pl-s">'energy'</span>: {
        <span class="pl-s">'total_energy'</span>: <span class="pl-s">'sum'</span>,
        <span class="pl-s">'energy_p98'</span>: <span class="pl-k">lambda</span> <span class="pl-s1">x</span>: <span class="pl-s1">np</span>.<span class="pl-en">percentile</span>(<span class="pl-s1">x</span>, <span class="pl-c1">98</span>),  <span class="pl-c"># lambda</span>
        <span class="pl-s">'energy_p17'</span>: <span class="pl-k">lambda</span> <span class="pl-s1">x</span>: <span class="pl-s1">np</span>.<span class="pl-en">percentile</span>(<span class="pl-s1">x</span>, <span class="pl-c1">17</span>),  <span class="pl-c"># lambda</span>
    },
    <span class="pl-s">'distance'</span>: {
        <span class="pl-s">'total_distance'</span>: <span class="pl-s">'sum'</span>,
        <span class="pl-s">'average_distance'</span>: <span class="pl-s">'mean'</span>,
        <span class="pl-s">'distance_mad'</span>: <span class="pl-s1">smrb</span>.<span class="pl-s1">mad</span>,   <span class="pl-c"># original function</span>
        <span class="pl-s">'distance_mad_c1'</span>: <span class="pl-s1">mad_c1</span>,  <span class="pl-c"># partial function wrapping the original function</span>
    },
})</pre></div>
<p dir="auto">results in</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="          energy                             distance
    total_energy energy_p98 energy_p17 total_distance average_distance distance_mad distance_mad_c1
cat
A           5.79     2.0364     1.8510           4.44            1.480     0.355825           0.240
B           2.85     1.5930     1.3095           1.83            0.915     0.140847           0.095
C           1.01     1.0100     1.0100           0.60            0.600     0.000000           0.000"><pre class="notranslate"><code class="notranslate">          energy                             distance
    total_energy energy_p98 energy_p17 total_distance average_distance distance_mad distance_mad_c1
cat
A           5.79     2.0364     1.8510           4.44            1.480     0.355825           0.240
B           2.85     1.5930     1.3095           1.83            0.915     0.140847           0.095
C           1.01     1.0100     1.0100           0.60            0.600     0.000000           0.000
</code></pre></div>
<p dir="auto">and all is left is:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="# get rid of the first MultiIndex level in a pretty straightforward way
mydf_agg.columns = mydf_agg.columns.droplevel(level=0)"><pre class="notranslate"><span class="pl-c"># get rid of the first MultiIndex level in a pretty straightforward way</span>
<span class="pl-s1">mydf_agg</span>.<span class="pl-s1">columns</span> <span class="pl-c1">=</span> <span class="pl-s1">mydf_agg</span>.<span class="pl-s1">columns</span>.<span class="pl-en">droplevel</span>(<span class="pl-s1">level</span><span class="pl-c1">=</span><span class="pl-c1">0</span>)</pre></div>
<p dir="auto">Happy dance praising pandas 💃 🕺 !</p>
<h3 dir="auto">After</h3>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import numpy as np
import statsmodels.robust as smrb
from functools import partial

# median absolute deviation as a partial function
# in order to demonstrate the issue with partial functions as aggregators
mad_c1 = partial(smrb.mad, c=1)

# no way of choosing the destination's column names...
mydf_agg = mydf.groupby('cat').agg({
    'energy': [
    	'sum',
    	lambda x: np.percentile(x, 98), # lambda
    	lambda x: np.percentile(x, 17), # lambda
    ],
    'distance': [
    	'sum',
    	'mean',
    	smrb.mad, # original function
    	mad_c1,   # partial function wrapping the original function
    ],
})"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">numpy</span> <span class="pl-k">as</span> <span class="pl-s1">np</span>
<span class="pl-k">import</span> <span class="pl-s1">statsmodels</span>.<span class="pl-s1">robust</span> <span class="pl-k">as</span> <span class="pl-s1">smrb</span>
<span class="pl-k">from</span> <span class="pl-s1">functools</span> <span class="pl-k">import</span> <span class="pl-s1">partial</span>

<span class="pl-c"># median absolute deviation as a partial function</span>
<span class="pl-c"># in order to demonstrate the issue with partial functions as aggregators</span>
<span class="pl-s1">mad_c1</span> <span class="pl-c1">=</span> <span class="pl-en">partial</span>(<span class="pl-s1">smrb</span>.<span class="pl-s1">mad</span>, <span class="pl-s1">c</span><span class="pl-c1">=</span><span class="pl-c1">1</span>)

<span class="pl-c"># no way of choosing the destination's column names...</span>
<span class="pl-s1">mydf_agg</span> <span class="pl-c1">=</span> <span class="pl-s1">mydf</span>.<span class="pl-en">groupby</span>(<span class="pl-s">'cat'</span>).<span class="pl-en">agg</span>({
    <span class="pl-s">'energy'</span>: [
    	<span class="pl-s">'sum'</span>,
    	<span class="pl-k">lambda</span> <span class="pl-s1">x</span>: <span class="pl-s1">np</span>.<span class="pl-en">percentile</span>(<span class="pl-s1">x</span>, <span class="pl-c1">98</span>), <span class="pl-c"># lambda</span>
    	<span class="pl-k">lambda</span> <span class="pl-s1">x</span>: <span class="pl-s1">np</span>.<span class="pl-en">percentile</span>(<span class="pl-s1">x</span>, <span class="pl-c1">17</span>), <span class="pl-c"># lambda</span>
    ],
    <span class="pl-s">'distance'</span>: [
    	<span class="pl-s">'sum'</span>,
    	<span class="pl-s">'mean'</span>,
    	<span class="pl-s1">smrb</span>.<span class="pl-s1">mad</span>, <span class="pl-c"># original function</span>
    	<span class="pl-s1">mad_c1</span>,   <span class="pl-c"># partial function wrapping the original function</span>
    ],
})</pre></div>
<p dir="auto">The above breaks because the lambda functions will all result in columns named <code class="notranslate">&lt;lambda&gt;</code> which results in</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="SpecificationError: Function names must be unique, found multiple named &lt;lambda&gt;"><pre class="notranslate"><code class="notranslate">SpecificationError: Function names must be unique, found multiple named &lt;lambda&gt;
</code></pre></div>
<p dir="auto"><strong>Backward incompatible regression: one cannot apply two different lambdas to the same original column anymore.</strong></p>
<p dir="auto">If one removes the <code class="notranslate">lambda x: np.percentile(x, 98)</code> from above, we get the same issue with the partial function which inherits the function name from the original function:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="SpecificationError: Function names must be unique, found multiple named mad"><pre class="notranslate"><code class="notranslate">SpecificationError: Function names must be unique, found multiple named mad
</code></pre></div>
<p dir="auto">Finally, after overwriting the <code class="notranslate">__name__</code> attribute of the partial (for example with <code class="notranslate">mad_c1.__name__ = 'mad_c1'</code>) we get:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="    energy          distance
       sum &lt;lambda&gt;      sum   mean       mad mad_c1
cat
A     5.79   1.8510     4.44  1.480  0.355825  0.240
B     2.85   1.3095     1.83  0.915  0.140847  0.095
C     1.01   1.0100     0.60  0.600  0.000000  0.000"><pre class="notranslate"><code class="notranslate">    energy          distance
       sum &lt;lambda&gt;      sum   mean       mad mad_c1
cat
A     5.79   1.8510     4.44  1.480  0.355825  0.240
B     2.85   1.3095     1.83  0.915  0.140847  0.095
C     1.01   1.0100     0.60  0.600  0.000000  0.000
</code></pre></div>
<p dir="auto">with still</p>
<ul dir="auto">
<li>one column missing (98th percentile)</li>
<li>the handling of the MultiIndex columns</li>
<li>and the renaming of the columns</li>
</ul>
<p dir="auto">to deal with in separate step.</p>
<p dir="auto">There is no control possible for the column names after aggregation, the best we can get in an automated way is some combination of original column name and the <em>aggregate function's name</em> like this:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="mydf_agg.columns = ['_'.join(col) for col in mydf_agg.columns]"><pre class="notranslate"><span class="pl-s1">mydf_agg</span>.<span class="pl-s1">columns</span> <span class="pl-c1">=</span> [<span class="pl-s">'_'</span>.<span class="pl-en">join</span>(<span class="pl-s1">col</span>) <span class="pl-k">for</span> <span class="pl-s1">col</span> <span class="pl-c1">in</span> <span class="pl-s1">mydf_agg</span>.<span class="pl-s1">columns</span>]</pre></div>
<p dir="auto">which results in:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="     energy_sum  energy_&lt;lambda&gt;  distance_sum  distance_mean  distance_mad distance_mad_c1
cat
A          5.79           1.8510          4.44          1.480      0.355825           0.240
B          2.85           1.3095          1.83          0.915      0.140847           0.095
C          1.01           1.0100          0.60          0.600      0.000000           0.000"><pre class="notranslate"><code class="notranslate">     energy_sum  energy_&lt;lambda&gt;  distance_sum  distance_mean  distance_mad distance_mad_c1
cat
A          5.79           1.8510          4.44          1.480      0.355825           0.240
B          2.85           1.3095          1.83          0.915      0.140847           0.095
C          1.01           1.0100          0.60          0.600      0.000000           0.000
</code></pre></div>
<p dir="auto">and if you really need to have different names, you can do it like this:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="mydf_agg.rename({
    &quot;energy_sum&quot;: &quot;total_energy&quot;,
    &quot;energy_&lt;lambda&gt;&quot;: &quot;energy_p17&quot;,
    &quot;distance_sum&quot;: &quot;total_distance&quot;,
    &quot;distance_mean&quot;: &quot;average_distance&quot;
    }, inplace=True)"><pre class="notranslate"><span class="pl-s1">mydf_agg</span>.<span class="pl-en">rename</span>({
    <span class="pl-s">"energy_sum"</span>: <span class="pl-s">"total_energy"</span>,
    <span class="pl-s">"energy_&lt;lambda&gt;"</span>: <span class="pl-s">"energy_p17"</span>,
    <span class="pl-s">"distance_sum"</span>: <span class="pl-s">"total_distance"</span>,
    <span class="pl-s">"distance_mean"</span>: <span class="pl-s">"average_distance"</span>
    }, <span class="pl-s1">inplace</span><span class="pl-c1">=</span><span class="pl-c1">True</span>)</pre></div>
<p dir="auto">but that means that you need to be careful to keep the renaming code (which must now be located at another place in the code) in sync with the code where the aggregation is defined...</p>
<p dir="auto">Sad pandas user <g-emoji class="g-emoji" alias="cry" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f622.png">😢</g-emoji>  (which still loves pandas of course)</p>
<hr>
<p dir="auto">I am all in for consistency, and at the same time I deeply regret the deprecation of the <em>aggregate and rename</em> functionality. I hope the examples above make the pain points clear.</p>
<hr>
<h3 dir="auto">Possible solutions</h3>
<ul dir="auto">
<li>Un-deprecate the dict-of-dict relabeling functionality</li>
<li>Provide another API to be able to do it (but why should there be two methods for the same main purpose, namely aggregation?)</li>
<li>??? (open to suggestions)</li>
</ul>
<hr>
<p dir="auto"><em>Optional read:</em></p>
<p dir="auto">With respect to the aforementioned discussion in the pull request which has been going on already for a few months, I only recently realized one of the reasons why I am so bothered by this deprecation: "aggregate and rename"  is a natural thing to do with GROUP BY aggregations in SQL since in SQL you usually provide the destination column name directly next to the aggregation expression, e.g. <code class="notranslate">SELECT col1, avg(col2) AS col2_mean, stddev(col2) AS col2_var FROM mytable GROUP BY col1</code>.</p>
<p dir="auto">I'm <strong><em>not</em></strong> saying that Pandas should necessarily provide the same functionalities as SQL of course. But the examples provided above demonstrate why the dict-of-dict API was in my opinion a clean and simple solution to many use-cases.</p>
<p dir="auto">(* I don't personally agree that the dict-of-dict approach is complex.)</p>
<p dir="auto"><code class="notranslate">astype/convert_objects</code> should be able to do this type of coercion from string complex looking</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="In [45]: Series(['1.0+5j','1.5-3j']).apply(lambda x: np.complex(x))
Out[45]: 
0      (1+5j)
1    (1.5-3j)
dtype: complex128"><pre class="notranslate"><code class="notranslate">In [45]: Series(['1.0+5j','1.5-3j']).apply(lambda x: np.complex(x))
Out[45]: 
0      (1+5j)
1    (1.5-3j)
dtype: complex128
</code></pre></div>
<p dir="auto"><a href="http://stackoverflow.com/questions/18919699/python-pandas-complex-number/18919965#18919965" rel="nofollow">http://stackoverflow.com/questions/18919699/python-pandas-complex-number/18919965#18919965</a></p>