title: Spark Save DataSet partitionby File Already Exists Error
body: <p>I have a DataSet which was constructed in the following way:</p>
  
  <pre><code> Encoder&lt;MyDomain&gt; encoder= Encoders.bean(MyDomain.class);
   Dataset&lt;MyDomain&gt; stdDS = sc.createDataset(filteredRecords.rdd(), encoder);
   Dataset&lt;Row&gt; rowDataset = stdDS.withColumn("idHash", stdDS.col("id").substr(0, 5));
  </code></pre>
  
  <p>I am then trying to output the dataset by doing:</p>
  
  <pre><code>        rowDataset.write().partitionBy("keep", "idHash").save("test.parquet");
  </code></pre>
  
  <p>When I only partition by "keep" everything works correctly, then I partition by both "keep" and "idHash" I get:</p>
  
  <pre><code>File already exists: file:/C:/dev/test.parquet/_temporary/0/_temporary/attempt_201701191219_0001_m_000000_0/keep=true/idHash=0a/part-r-00000-2c2e0494-f6a7-47d7-88e2-f49dffb608d1.snappy.parquet
  </code></pre>
  
  <p>How can I get my DataSet to properly output using multiple partitions.  The folder is empty to start with.  Also this error is happening when I run on my local machine, in production this data will be output to S3 so any solution needs to work both against a local filesystem and AWS S3.</p>
  
  <p>Thanks,
  Nathan</p>