title: Calculating autocorrelation in spark via sparklyr generates inconsistent results
body: <p>I am trying to calculate the autocorrelation within one column of a spark dataframe using sparklyr and <code>spark_apply()</code> using the following code snippet.</p>
  
  <pre><code># Read raw data
  sdf_raw &lt;- spark_read_parquet(sc = sc, name = "sdf_raw", paste0(root_path, "sdf_raw_clean"))
  
  # Get subset containing only relevant column and cache the result
  t &lt;- sdf_raw %&gt;% select(src_ssn) %&gt;% repartition(1) %&gt;% compute("subset")
  
  # Apply acf function via spark_apply
  q &lt;- spark_apply(t, function(data) {
    tmp = acf(data$src_ssn, lag.max = 10, plot = F)
    res = data.frame(acf = tmp$acf, lag = seq(0, length(tmp$acf)-1))
    res
  })
  
  # Collect the result
  e &lt;- q %&gt;% collect()
  </code></pre>
  
  <p>The desired result would be something like</p>
  
  <pre><code>            acf lag
  1   1.000000000   0
  2   0.091755960   1
  3   0.043182701   2
  4   0.012342259   3
  5  -0.009166059   4
  6  -0.020663184   5
  7  -0.026083233   6
  8  -0.026233158   7
  9  -0.021264385   8
  10 -0.014011600   9
  11 -0.009580071  10
  </code></pre>
  
  <p>However, the resulting spark dataframe contains additional rows repeating the lag sequence</p>
  
  <pre><code># A tibble: 3,201 x 2
           acf   lag
         &lt;dbl&gt; &lt;int&gt;
   1  1            0
   2  0.0993       1
   3  0.0448       2
   4  0.00947      3
   5 -0.00656      4
   6 -0.0122       5
   7 -0.0198       6
   8 -0.00980      7
   9 -0.000583     8
  10 -0.0148       9
  11 -0.0234       10
  12  1            0
  13 -0.4          1
  14 -0.53         2
  # … with 3,191 more rows
  </code></pre>
  
  <p>I assumed this has something to do with the partitioning of the data between executors. However, repartitioning <code>t</code> to a single partition using <code>repartition(1)</code> has not solved the problem. Is this something that is not possible this way or is it something I am doing wrong?</p>