title: Configure AWS EMR spark with aws sts assume-role
body: <p>I am trying to read a file from s3 bucket with is in another AWS Account.</p>
  
  <p>Steps:</p>
  
  <ol>
  <li>Generated access id, access key secret and token using <strong><code>aws sts assume-role</code></strong> command</li>
  <li><p>Set following variables:</p>
  
  <pre><code>export AWS_ACCESS_KEY_ID=
  export AWS_SECRET_ACCESS_KEY=
  export AWS_SESSION_TOKEN=
  </code></pre></li>
  <li><p><code>aws s3 ls s3://...</code> </p></li>
  <li><p>reading file in <strong>spark-shell</strong></p>
  
  <pre><code>sc.hadoopConfiguration.set("fs.s3n.awsAccessKeyId", "XXX")
  sc.hadoopConfiguration.set("fs.s3n.awsSecretAccessKey", "XXX")
  sc.hadoopConfiguration.set("fs.s3a.session.token", "XXX")
  spark.read.parquet("file location")
  </code></pre></li>
  </ol>
  
  <p>Up to step 3 its working fine. </p>
  
  <p>I am unable to read the file in spark-shell.
  I am using <strong>spark version: 2.4.3</strong>. Please let me know what i am missing. Any help would be appreciated.</p>
  
  <p><strong>Error log:</strong> </p>
  
  <pre><code>    Caused by: com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.model.AmazonS3Exception: The AWS Access Key Id you provided does not exist in our records. 
  </code></pre>