title: Extremely slow custom function in pandas aggregate
body: <p>I have to calculate the coefficient of variation (std/ mean) in a moving window in a dataset I have. The code is pretty simple:</p>
  <pre><code>start_time = time.time()
  
  channel_data = data[data['channel_id'] == ch].sort_values('datetime')
  channel_data['datetime'] = pd.to_datetime(channel_data['datetime'],format='%Y-%m-%d %H:%M:%S').dt.floor('d')
  channel_data = channel_data.set_index('datetime')
  
  rolling_data = channel_data.rolling('{}D'.format(window)).agg({ 'val': coeff_of_var })
  
  rolling_data = rolling_data.groupby(rolling_data.index.floor('d')).last()[window:]
  
  print('Time needed: {} seconds'.format(time.time() - start_time))
  </code></pre>
  <p>My problem is that it's extremely slow.</p>
  <p><code>coeff_of_var</code> can either be a custom function of <code>scipy.stats.variation</code> or a lambda function - I tried them all. The custom function I use is</p>
  <pre><code>def coeff_of_var(series):
      return series.std() / series.mean()
  </code></pre>
  <p><code>data</code> is a DataFrame with dimensions [3070584 rows x 5 columns]. <code>channel_data</code> contains ~457710 rows.</p>
  <p>This code is inside a cycle and it takes so long that I had to stop it before the <code>print</code> of the first execution time...</p>
  <p>Instead of calculating the coefficient of variation, I tried calculating separately both std and mean, which is, to substitute the function in the curly brackets of line 5 with: ['std', 'mean']. The new line is:</p>
  <pre><code>std_rolling_data = channel_data.rolling('{}D'.format(window)).agg({ 'val': ['std', 'mean'] })
  </code></pre>
  <p>The execution time drops to:</p>
  <pre><code>Time needed: 0.9421329498291016 seconds
  Time needed: 0.9423763751983643 seconds
  Time needed: 0.9420042037963867 seconds
  Time needed: 0.9560058116912842 seconds
  Time needed: 0.9728689193725586 seconds
  </code></pre>
  <p>I also tried simplifying the custom function to:</p>
  <pre><code>def coeff_of_var(series):
      return 1
  </code></pre>
  <p>In this case the execution time is:</p>
  <pre><code>Time needed: 32.465901136398315 seconds
  Time needed: 34.14194059371948 seconds
  Time needed: 38.883220195770264 seconds
  Time needed: 44.430686950683594 seconds
  Time needed: 42.99840021133423 seconds
  </code></pre>
  <p>Do you have any ideas what's the cause of the slowness with the custom function?</p>
  <hr />
  <h1>EDIT</h1>
  <p>You can replicate the problem with the following code:</p>
  <pre><code>import numpy as np
  import pandas as pd
  
  import time
  
  def coeff_of_var(series):
      return 1 #series.std() / series.mean()
  
  nrows = 3070584
  ntrue = 450000
  window = 5
  
  rng = pd.date_range('2015-01-01 00:00:00', periods=nrows, freq='min')
  data = pd.DataFrame({ 'A': rng, 'B' : np.random.randn(len(rng)), 'C' : np.random.randn(len(rng)), 'D' : np.random.randn(len(rng)), 'E' : np.zeros(len(rng), dtype=bool)})
  
  data.loc[np.random.choice(np.arange(nrows), ntrue, replace=False), 'E'] = True
  
  start_time = time.time()
  
  channel_data = data[data['E'] == True].sort_values('A')
  channel_data['A'] = pd.to_datetime(channel_data['A'],format='%Y-%m-%d %H:%M:%S').dt.floor('d')
  channel_data = channel_data.set_index('A')
  
  rolling_data = channel_data.rolling('{}D'.format(window)).agg({ 'B': coeff_of_var })
  
  rolling_data = rolling_data.groupby(rolling_data.index.floor('d')).last()[window:]
  
  print('Time needed: {} seconds'.format(time.time() - start_time))
  </code></pre>