title: How to undersampling the majority class using pyspark
body: <p>I try to solve the data like below code,but I have not figured it out using groupy and udf ,and also find udf can not return dataframe.</p>
  
  <p>Is there any way that could implement this by spark or some other method could handle unbalanced data</p>
  
  <p><div class="snippet" data-lang="js" data-hide="false" data-console="true" data-babel="false">
  <div class="snippet-code">
  <pre class="snippet-code-html lang-html prettyprint-override"><code>ratio = 3
  def balance_classes(grp):
      picked = grp.loc[grp.editorsSelection == True]
      n = round(picked.shape[0]*ratio)
      if n:        
          try:
              not_picked = grp.loc[grp.editorsSelection == False].sample(n)
          except: # In case, fewer than n comments with `editorsSelection == False`
              not_picked = grp.loc[grp.editorsSelection == False]
          balanced_grp = pd.concat([picked, not_picked])
          return balanced_grp
      else: # If no editor's pick for an article, dicard all comments from that article
          return None 
  
  comments = comments.groupby('articleID').apply(balance_classes).reset_index(drop=True)</code></pre>
  </div>
  </div>
  </p>