title: How to apply a custom function to groups in a dask dataframe, using multiple columns as function input
body: <p>I have a very large dataframe that I'm handling with <strong>dask</strong>. 
  The dataframe looks by and large like this:</p>
  
  <pre><code>Col_1    Col_2   Bool_1   Bool_2
  A        1       True     False
  B        1       True     True
  C        1       False    False
  D        1       True     False
  A        2       False    True
  B        2       False    False
  C        2       True     False
  D        2       True     True
  </code></pre>
  
  <p>But it has millions of rows. </p>
  
  <p>What I'm trying to do at this point of the code is to calculate a <strong>Jaccard distance</strong> between <code>Bool_1</code> and <code>Bool_2</code> for each group formed in <code>Col_2</code>. This, because the aim of this program is to produce <em>one line for each group</em> that is present in <code>Col_2</code> (each line has several statistics, I'm reporting only the relevant columns).</p>
  
  <p>To do so, I first group the dataframe by <code>Col_2</code> using <code>df.groupby("Col_2")</code>, but then I don't know how to proceed. Every attempt I tried so far has thrown an error.</p>
  
  <p><strong>1</strong>: I tried to define a function <code>compute_jacc_dist()</code> and to pass it via <code>apply(compute_jacc_dist, axis=1)</code> to the groups, but it has issues with the args and kwargs (the axis especially, see <a href="https://github.com/dask/dask/issues/1572" rel="nofollow noreferrer">https://github.com/dask/dask/issues/1572</a> , which I couldn't solve yet).</p>
  
  <p><strong>2</strong>: I tried to use <code>from dask_distance import jaccard</code> and use it to compute the J distance between <code>Bool_1</code> and <code>Bool_2</code> but it produces weird results (each group returns J=1 even if there is NO intersection). </p>
  
  <p><strong>3</strong>: I tried to <code>compute()</code> the dataframe and to iterate over the groups using: </p>
  
  <pre><code>for name, group in df.groupby("Col_2"):
     jacc = dask_distance.jaccard(group["Bool_1"], group["Bool_2"])
  </code></pre>
  
  <p>But this one is slow as hell because it triggers a computation and then operates over such a huge dataframe group by group (i.e. I don't want to use it). For reference, a script with this function is running since two days, while I estimate that any of the solutions #1 and #2 I have tried, if properly set, would return results in 1-2 hours. </p>
  
  <p>Any suggestion on how I could handle this issue? My ideal solution would be to use <code>df.groupby("Col_1").apply(compute_jacc_dist)</code> in a proper way. Any help much appreciated! </p>