<p dir="auto">Today <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/shoyer/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/shoyer">@shoyer</a> and I were talking about a new "protocol" that will let us sidestep the whole macro / method chaining issue.  The basic idea is that pandas objects define a <code class="notranslate">pipe</code> method (ideally other libraries will implement this to, assuming this is useful).</p>
<p dir="auto">Based on the discussions below, we're leaning towards a method like</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="def pipe(self, func, *args, **kwargs):
    pipe_func = getattr(func, '__pipe_func__', func)
    return pipe_func(self, *args, **kwargs)"><pre class="notranslate"><span class="pl-k">def</span> <span class="pl-en">pipe</span>(<span class="pl-s1">self</span>, <span class="pl-s1">func</span>, <span class="pl-c1">*</span><span class="pl-s1">args</span>, <span class="pl-c1">**</span><span class="pl-s1">kwargs</span>):
    <span class="pl-s1">pipe_func</span> <span class="pl-c1">=</span> <span class="pl-en">getattr</span>(<span class="pl-s1">func</span>, <span class="pl-s">'__pipe_func__'</span>, <span class="pl-s1">func</span>)
    <span class="pl-k">return</span> <span class="pl-en">pipe_func</span>(<span class="pl-s1">self</span>, <span class="pl-c1">*</span><span class="pl-s1">args</span>, <span class="pl-c1">**</span><span class="pl-s1">kwargs</span>)</pre></div>
<p dir="auto">That's it. This lets you write code like:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import seaborn as sns

(sns.load_dataset('iris')
    .query(&quot;sepal_length &gt; 5&quot;)
    .assign(sepal_ratio = lambda df: df.sepal_length / df.sepal_width)
    .pipe(lambda df: sns.violinplot(x='species', y='sepal_ratio', data=df))
)"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">seaborn</span> <span class="pl-k">as</span> <span class="pl-s1">sns</span>

(<span class="pl-s1">sns</span>.<span class="pl-en">load_dataset</span>(<span class="pl-s">'iris'</span>)
    .<span class="pl-en">query</span>(<span class="pl-s">"sepal_length &gt; 5"</span>)
    .<span class="pl-en">assign</span>(<span class="pl-s1">sepal_ratio</span> <span class="pl-c1">=</span> <span class="pl-k">lambda</span> <span class="pl-s1">df</span>: <span class="pl-s1">df</span>.<span class="pl-s1">sepal_length</span> <span class="pl-c1">/</span> <span class="pl-s1">df</span>.<span class="pl-s1">sepal_width</span>)
    .<span class="pl-en">pipe</span>(<span class="pl-k">lambda</span> <span class="pl-s1">df</span>: <span class="pl-s1">sns</span>.<span class="pl-en">violinplot</span>(<span class="pl-s1">x</span><span class="pl-c1">=</span><span class="pl-s">'species'</span>, <span class="pl-s1">y</span><span class="pl-c1">=</span><span class="pl-s">'sepal_ratio'</span>, <span class="pl-s1">data</span><span class="pl-c1">=</span><span class="pl-s1">df</span>))
)</pre></div>
<p dir="auto">seaborn didn't have to do anything! If the DataFrame is the first argument to the function, things are even simpler:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="def f(x):
    return x / 2

df['sepal_length'].pipe(f)"><pre class="notranslate"><span class="pl-k">def</span> <span class="pl-en">f</span>(<span class="pl-s1">x</span>):
    <span class="pl-k">return</span> <span class="pl-s1">x</span> <span class="pl-c1">/</span> <span class="pl-c1">2</span>

<span class="pl-s1">df</span>[<span class="pl-s">'sepal_length'</span>].<span class="pl-en">pipe</span>(<span class="pl-s1">f</span>)</pre></div>
<p dir="auto">Users or libraries can work around the need for the (somewhat ugly) <code class="notranslate">lambda _:</code>, by using the <code class="notranslate">__pipe_func__</code>  attribute of the function being <code class="notranslate">pipe</code>d in. This is where a protocol (de facto or official) would be useful, since libraries that know nothing else about each other can rely on it. As an example, consider seaborn's violin plot, which expects a DataFrame as its fourth argument, <code class="notranslate">data</code>. Seaborn can define a simple decorator to attach a <code class="notranslate">__pipe_func__</code> attribute, allowing <em>it</em> to define how it expects to be <code class="notranslate">pipe</code>d to.</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="def pipeable(func):
    def pipe_func(data, *args, **kwargs):
        return func(*args, data=data, **kwargs)
    func.__pipe_func__ = pipe_func
    return func

# now we decorate all the Seaborn methods as pipeable

@pipeable
def violinplot(x=None, y=None, hue=None, data=None, ...):
    # ...   "><pre class="notranslate"><span class="pl-k">def</span> <span class="pl-en">pipeable</span>(<span class="pl-s1">func</span>):
    <span class="pl-k">def</span> <span class="pl-en">pipe_func</span>(<span class="pl-s1">data</span>, <span class="pl-c1">*</span><span class="pl-s1">args</span>, <span class="pl-c1">**</span><span class="pl-s1">kwargs</span>):
        <span class="pl-k">return</span> <span class="pl-en">func</span>(<span class="pl-c1">*</span><span class="pl-s1">args</span>, <span class="pl-s1">data</span><span class="pl-c1">=</span><span class="pl-s1">data</span>, <span class="pl-c1">**</span><span class="pl-s1">kwargs</span>)
    <span class="pl-s1">func</span>.<span class="pl-s1">__pipe_func__</span> <span class="pl-c1">=</span> <span class="pl-s1">pipe_func</span>
    <span class="pl-k">return</span> <span class="pl-s1">func</span>

<span class="pl-c"># now we decorate all the Seaborn methods as pipeable</span>

<span class="pl-en">@<span class="pl-s1">pipeable</span></span>
<span class="pl-k">def</span> <span class="pl-en">violinplot</span>(<span class="pl-s1">x</span><span class="pl-c1">=</span><span class="pl-c1">None</span>, <span class="pl-s1">y</span><span class="pl-c1">=</span><span class="pl-c1">None</span>, <span class="pl-s1">hue</span><span class="pl-c1">=</span><span class="pl-c1">None</span>, <span class="pl-s1">data</span><span class="pl-c1">=</span><span class="pl-c1">None</span>, ...):
    <span class="pl-c"># ...   </span></pre></div>
<p dir="auto">And users write</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="(sns.load_dataset('iris')
    .query(&quot;sepal_length &gt; 5&quot;)
    .assign(sepal_ratio = lambda x: x.sepal_length / x.sepal_width)
    .pipe(sns.violinplot, x='species', y='sepal_ratio')
)"><pre class="notranslate">(<span class="pl-s1">sns</span>.<span class="pl-en">load_dataset</span>(<span class="pl-s">'iris'</span>)
    .<span class="pl-en">query</span>(<span class="pl-s">"sepal_length &gt; 5"</span>)
    .<span class="pl-en">assign</span>(<span class="pl-s1">sepal_ratio</span> <span class="pl-c1">=</span> <span class="pl-k">lambda</span> <span class="pl-s1">x</span>: <span class="pl-s1">x</span>.<span class="pl-s1">sepal_length</span> <span class="pl-c1">/</span> <span class="pl-s1">x</span>.<span class="pl-s1">sepal_width</span>)
    .<span class="pl-en">pipe</span>(<span class="pl-s1">sns</span>.<span class="pl-s1">violinplot</span>, <span class="pl-s1">x</span><span class="pl-c1">=</span><span class="pl-s">'species'</span>, <span class="pl-s1">y</span><span class="pl-c1">=</span><span class="pl-s">'sepal_ratio'</span>)
)</pre></div>
<h3 dir="auto">Why?</h3>
<p dir="auto">Heavily nested function calls are bad. They're hard to read, and can easily introduce bugs. Consider:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="# f, g, and h are functions that take and receive a DataFrame
result = f(g(h(df), arg1=1), arg2=2, arg3=3)"><pre class="notranslate"><span class="pl-c"># f, g, and h are functions that take and receive a DataFrame</span>
<span class="pl-s1">result</span> <span class="pl-c1">=</span> <span class="pl-en">f</span>(<span class="pl-en">g</span>(<span class="pl-en">h</span>(<span class="pl-s1">df</span>), <span class="pl-s1">arg1</span><span class="pl-c1">=</span><span class="pl-c1">1</span>), <span class="pl-s1">arg2</span><span class="pl-c1">=</span><span class="pl-c1">2</span>, <span class="pl-s1">arg3</span><span class="pl-c1">=</span><span class="pl-c1">3</span>)</pre></div>
<p dir="auto">For pandas, the approach has been to add <code class="notranslate">f</code>, <code class="notranslate">g</code>, and <code class="notranslate">h</code> as methods to (say) <code class="notranslate">DataFrame</code></p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="(df.h()
   .g(arg1=1)
   .f(arg2=2, arg3=3)
)"><pre class="notranslate">(<span class="pl-s1">df</span>.<span class="pl-en">h</span>()
   .<span class="pl-en">g</span>(<span class="pl-s1">arg1</span><span class="pl-c1">=</span><span class="pl-c1">1</span>)
   .<span class="pl-en">f</span>(<span class="pl-s1">arg2</span><span class="pl-c1">=</span><span class="pl-c1">2</span>, <span class="pl-s1">arg3</span><span class="pl-c1">=</span><span class="pl-c1">3</span>)
)</pre></div>
<p dir="auto">The code is certainly cleaner. It reads and flows top to bottom instead of inside-out. The function arguments are next to the function calls. But there's a hidden cost. DataFrame has something like 200+ methods, which is crazy. It's less flexible for users since it's hard to get their own functions into pipelines (short of monkey-patching). With <code class="notranslate">.pipe</code>, we can</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="(df.pipe(h)
   .pipe(g, arg1=1)
   .pipe(f, arg2=2, arg3=3)
)"><pre class="notranslate">(<span class="pl-s1">df</span>.<span class="pl-en">pipe</span>(<span class="pl-s1">h</span>)
   .<span class="pl-en">pipe</span>(<span class="pl-s1">g</span>, <span class="pl-s1">arg1</span><span class="pl-c1">=</span><span class="pl-c1">1</span>)
   .<span class="pl-en">pipe</span>(<span class="pl-s1">f</span>, <span class="pl-s1">arg2</span><span class="pl-c1">=</span><span class="pl-c1">2</span>, <span class="pl-s1">arg3</span><span class="pl-c1">=</span><span class="pl-c1">3</span>)
)</pre></div>
<p dir="auto">The other way around the nested calls is using temporary variables:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="r1 = h(df)
r2 = g(r1, arg1=1)
r3 = f(r2, arg2=2, arg3=3)"><pre class="notranslate"><span class="pl-s1">r1</span> <span class="pl-c1">=</span> <span class="pl-en">h</span>(<span class="pl-s1">df</span>)
<span class="pl-s1">r2</span> <span class="pl-c1">=</span> <span class="pl-en">g</span>(<span class="pl-s1">r1</span>, <span class="pl-s1">arg1</span><span class="pl-c1">=</span><span class="pl-c1">1</span>)
<span class="pl-s1">r3</span> <span class="pl-c1">=</span> <span class="pl-en">f</span>(<span class="pl-s1">r2</span>, <span class="pl-s1">arg2</span><span class="pl-c1">=</span><span class="pl-c1">2</span>, <span class="pl-s1">arg3</span><span class="pl-c1">=</span><span class="pl-c1">3</span>)</pre></div>
<p dir="auto">Which is better, but not as good as the <code class="notranslate">.pipe</code> solution.</p>
<hr>
<p dir="auto">A relevant thread on python-ideas, started by <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/mrocklin/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/mrocklin">@mrocklin</a>: <a href="https://mail.python.org/pipermail/python-ideas/2015-March/032745.html" rel="nofollow">https://mail.python.org/pipermail/python-ideas/2015-March/032745.html</a></p>
<p dir="auto">This doesn't achieve everything macros could. We still can't do things like <code class="notranslate">df.plot(x=x_col, y=y_col)</code> where <code class="notranslate">x_col</code> and <code class="notranslate">y_col</code> are captured by <code class="notranslate">df</code>'s namespace. But it may be good enough.</p>
<p dir="auto">Going to cc a bunch of people here, who've had interest in the past.</p>
<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/shoyer/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/shoyer">@shoyer</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/mrocklin/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/mrocklin">@mrocklin</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/datnamer/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/datnamer">@datnamer</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/dalejung/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/dalejung">@dalejung</a></p>
<p dir="auto">I mentioned this originally on <a href="https://groups.google.com/forum/#!topic/pydata/woxbB8na3C8" rel="nofollow">https://groups.google.com/forum/#!topic/pydata/woxbB8na3C8</a>.</p>
<p dir="auto">ewma() simply ignores missing values, effectively calculating the exponentially weighted moving average on the compacted series without the missing values. I think this is incorrect, and that values should be weighted based on their absolute location.</p>
<p dir="auto">In the code below, I reproduce the ewma() calculation using "wrong" weights, and then show what I believe the correct result should be using the "right" weights.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="In [1]: from pandas import Series, ewma

In [2]: def simple_wma(x, w):
   ...:     return x.multiply(w).cumsum() / w.cumsum()
   ...:

In [3]: s = Series([0, None, 100])

In [4]: com = 2

In [5]: alpha = 1/(1+com)

In [6]: wrong_weights_adjust_false = Series([(1-alpha), None, alpha])

In [7]: wrong_weights_adjust_true = Series([(1-alpha), None, 1])

In [8]: right_weights_adjust_false = Series([(1-alpha)**2, None, alpha])

In [9]: right_weights_adjust_true = Series([(1-alpha)**2, None, 1])

In [10]: ewma(s, com=com, adjust=False)
Out[10]:
0     0.000000
1     0.000000
2    33.333333
dtype: float64

In [11]: simple_wma(s, wrong_weights_adjust_false)
Out[11]:
0     0.000000
1          NaN
2    33.333333
dtype: float64

In [12]: simple_wma(s, right_weights_adjust_false)
Out[12]:
0     0.000000
1          NaN
2    42.857143
dtype: float64

In [13]: ewma(s, com=com, adjust=True)
Out[13]:
0     0
1     0
2    60
dtype: float64

In [14]: simple_wma(s, wrong_weights_adjust_true)
Out[14]:
0     0
1   NaN
2    60
dtype: float64

In [15]: simple_wma(s, right_weights_adjust_true)
Out[15]:
0     0.000000
1          NaN
2    69.230769
dtype: float64

In [16]: s2 = Series([0, 100])

In [17]: ewma(s2, com=com, adjust=False)
Out[17]:
0     0.000000
1    33.333333
dtype: float64

In [18]: ewma(s2, com=com, adjust=True)
Out[18]:
0     0
1    60
dtype: float64"><pre class="notranslate"><code class="notranslate">In [1]: from pandas import Series, ewma

In [2]: def simple_wma(x, w):
   ...:     return x.multiply(w).cumsum() / w.cumsum()
   ...:

In [3]: s = Series([0, None, 100])

In [4]: com = 2

In [5]: alpha = 1/(1+com)

In [6]: wrong_weights_adjust_false = Series([(1-alpha), None, alpha])

In [7]: wrong_weights_adjust_true = Series([(1-alpha), None, 1])

In [8]: right_weights_adjust_false = Series([(1-alpha)**2, None, alpha])

In [9]: right_weights_adjust_true = Series([(1-alpha)**2, None, 1])

In [10]: ewma(s, com=com, adjust=False)
Out[10]:
0     0.000000
1     0.000000
2    33.333333
dtype: float64

In [11]: simple_wma(s, wrong_weights_adjust_false)
Out[11]:
0     0.000000
1          NaN
2    33.333333
dtype: float64

In [12]: simple_wma(s, right_weights_adjust_false)
Out[12]:
0     0.000000
1          NaN
2    42.857143
dtype: float64

In [13]: ewma(s, com=com, adjust=True)
Out[13]:
0     0
1     0
2    60
dtype: float64

In [14]: simple_wma(s, wrong_weights_adjust_true)
Out[14]:
0     0
1   NaN
2    60
dtype: float64

In [15]: simple_wma(s, right_weights_adjust_true)
Out[15]:
0     0.000000
1          NaN
2    69.230769
dtype: float64

In [16]: s2 = Series([0, 100])

In [17]: ewma(s2, com=com, adjust=False)
Out[17]:
0     0.000000
1    33.333333
dtype: float64

In [18]: ewma(s2, com=com, adjust=True)
Out[18]:
0     0
1    60
dtype: float64
</code></pre></div>