title: Failed to rename S3AFileStatus
body: <p>I am getting following error when running on local EC2 with 'hadoop-3.1.1' and 'spark-2.4.0-bin-without-hadoop'.</p>
  
  <pre><code>[Stage 6:&gt;                                                       (0 + 32) / 200]
  2019-10-03 15:40:34 ERROR SparkHadoopMapRedUtil:91 - Error committing the output of task: attempt_20191003154033_0006_m_000000_0
  java.io.IOException: 
  Failed to rename S3AFileStatus{path=s3a://mck-547-data/data_dev/z_ref/ref_proc/_temporary/0/_temporary/attempt_20191003154033_0006_m_000000_0/part-00000-f3c3b55e-97e9-47d1-a7e2-1b35177e6206-c000.snappy.parquet; 
  isDirectory=false; length=7945; replication=1; blocksize=33554432; modification_time=1570117234000; access_time=0; owner=kk_shinde; group=kk_shinde; permission=rw-rw-rw-; isSymlink=false; hasAcl=false; isEncrypted=false; isErasureCoded=false} isEmptyDirectory=FALSE 
  to s3a://mck-547-data/data_dev/z_ref/ref_proc/part-00000-f3c3b55e-97e9-47d1-a7e2-1b35177e6206-c000.snappy.parquet
  </code></pre>
  
  <p>I googled and all the links re committer as below.</p>
  
  <p><a href="https://issues.apache.org/jira/browse/HADOOP-14161" rel="nofollow noreferrer">https://issues.apache.org/jira/browse/HADOOP-14161</a></p>
  
  <p><a href="https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/committers.md" rel="nofollow noreferrer">https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/committers.md</a></p>
  
  <p>Tried the option of adding the below properties to spark-conf:</p>
  
  <pre><code>#spark.hadoop.fs.s3a.buffer.dir /tmp/spark_tmp/
  #spark.hadoop.s.s3a.committer.staging.conflict-mode replace
  </code></pre>
  
  <p>But still getting this error. Just wondering what would be the easy solution. We are trying for an option of writing to local and doing S3 sync in the code. But that is very crude way of doing it.</p>
  
  <p>Any suggestions?</p>