title: pyspark using one task for mapPartitions when converting rdd to dataframe
body: <p>I'm confused as to why it appears that Spark is using 1 task for <code>rdd.mapPartitions</code> when converting the resulting RDD to a DataFrame. </p>
  
  <p>This is an issue for me because I would like to go from :</p>
  
  <p><code>DataFrame</code> --> <code>RDD</code> --> <code>rdd.mapPartitions</code> --> <code>DataFrame</code> </p>
  
  <p>so that I can read in data (DataFrame), apply a non-SQL function to chunks of data (mapPartitions on RDD) and then convert back to a DataFrame so that I can using the <code>DataFrame.write</code> process. </p>
  
  <p>I am able to go from DataFrame --> mapPartitions and then use an RDD writer like saveAsTextFile but that is less than ideal since the <code>DataFrame.write</code> process can do things like overwrite and save data in Orc format. So I'd like to learn <em>why</em> this is going on, but from a pratical perspective I'm primarily concerned with being able to just go from a DataFrame --> mapParitions --> to using the DataFrame.write process.</p>
  
  <p>Here is a reproducible example. The following works as expected, with 100 tasks for the <code>mapPartitions</code> work:</p>
  
  <pre><code>from pyspark.sql import SparkSession
  import pandas as pd
  
  spark = SparkSession \
      .builder \
      .master("yarn-client") \
      .enableHiveSupport() \
      .getOrCreate()
  
  sc = spark.sparkContext
  
  df = pd.DataFrame({'var1':range(100000),'var2': [x-1000 for x in range(100000)]})
  spark_df = spark.createDataFrame(df).repartition(100)
  
  def f(part):
      return [(1,2)]
  
  spark_df.rdd.mapPartitions(f).collect()
  </code></pre>
  
  <p>However if the last line is change to something like <code>spark_df.rdd.mapPartitions(f).toDF().show()</code> then there will only be one task for the <code>mapPartitions</code> work.</p>
  
  <p>Some screenshots illustrating this below:
  <a href="https://i.stack.imgur.com/g2sn0.png" rel="noreferrer"><img src="https://i.stack.imgur.com/g2sn0.png" alt="enter image description here"></a>
  <a href="https://i.stack.imgur.com/CoeNX.png" rel="noreferrer"><img src="https://i.stack.imgur.com/CoeNX.png" alt="enter image description here"></a></p>