title: Spark SQL dataframe.save with partitionBy is creating an array column
body: <p>I am trying to save the data of a Spark SQL dataframe to hive. The data that is to be stored should be partitioned by one of the columns in the dataframe. For that I have written the following code.</p>
  
  <pre><code>val conf = new SparkConf().setAppName("Hive partitioning")
  conf.set("spark.scheduler.mode", "FAIR")
  val sc = new SparkContext(conf)
  val hiveContext = new HiveContext(sc)
  hiveContext.setConf("hive.exec.dynamic.partition", "true")
  hiveContext.setConf("hive.exec.dynamic.partition.mode", "nonstrict")
  
  val df = hiveContext.sql("....   my sql query ....")
  
  df.printSchema()
  df.write.mode(SaveMode.Append).partitionBy("&lt;partition column&gt;").saveAsTable("orgs_partitioned")
  </code></pre>
  
  <p>The dataframe is getting stored as table with single column called <code>col</code> and of type <code>array&lt;string&gt;</code>, the structure is as shown below(Screenshot from Hue).</p>
  
  <p><a href="https://i.stack.imgur.com/5ZqMs.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/5ZqMs.png" alt="image from hue"></a></p>
  
  <p>Any pointers are very helpful.
  Thanks.</p>