title: Applying a function in each row of a big PySpark dataframe?
body: <p>I have a big dataframe (~30M rows). I have a function <code>f</code>. The business of <code>f</code> is to run through each row, check some logics and feed the outputs into a dictionary. The function needs to be performed row by row.</p>
  <p>I tried:</p>
  <p><code>dic = dict() for row in df.rdd.collect(): f(row, dic)</code></p>
  <p>But I always meet the error OOM. I set the memory of Docker to 8GB.</p>
  <p>How can I effectively perform the business?</p>