title: Returning a dataframe in Dask
body: <p><strong>Aim:</strong> To speed up applying a function row wise across a large data frame (1.9 million ~ rows) </p>
  
  <p><strong>Attempt:</strong> Using dask map_partitions where partitions == number of cores. I've written a function which is applied to each row, creates a dict containing a variable number of new values (between 1 and 55). This function works fine standalone. </p>
  
  <p><strong>Problem:</strong> I need a way to combine the output of each function into a final dataframe. I tried using df.append, where I'd append each dict to a new dataframe and return this dataframe. If I understand the Dask Docs, Dask should then combine them to one big DF. Unfortunately this line is tripping an error (<em>ValueError: could not broadcast input array from shape (56) into shape (1)</em>). Which leads me to believe it's something to do with the combine feature in Dask? </p>
  
  <pre><code>#Function to applied row wise down the dataframe. Takes a column (post) and new empty df. 
  def func(post,New_DF):
      post = str(post)
      scores = OtherFUNC.countWords(post)
      scores['post'] = post
      New_DF = New_DF.append(scores, ignore_index=True)
      return(New_DF)
  
  #Dask 
  dd.from_pandas(dataset,npartitions=nCores).\
   map_partitions(
        lambda df : df.apply(
           lambda x : func(x.post,New_DF),axis=1)).\
     compute(get=get)
  </code></pre>