title: Spark History Server on S3A FileSystem: ClassNotFoundException
body: <p>Spark can use Hadoop S3A file system <code>org.apache.hadoop.fs.s3a.S3AFileSystem</code>. By adding the following into the <code>conf/spark-defaults.conf</code>, I can get spark-shell to log to the S3 bucket:</p>
  
  <pre><code>spark.jars.packages               net.java.dev.jets3t:jets3t:0.9.0,com.google.guava:guava:16.0.1,com.amazonaws:aws-java-sdk:1.7.4,org.apache.hadoop:hadoop-aws:2.7.3
  spark.hadoop.fs.s3a.impl          org.apache.hadoop.fs.s3a.S3AFileSystem
  spark.eventLog.enabled            true
  spark.eventLog.dir                s3a://spark-logs-test/
  spark.history.fs.logDirectory     s3a://spark-logs-test/
  spark.history.provider            org.apache.hadoop.fs.s3a.S3AFileSystem
  </code></pre>
  
  <p>Spark History Server also loads configuration from <code>conf/spark-defaults.conf</code>, but it seems not to load <code>spark.jars.packages</code> configuration, and throws <code>ClassNotFoundException</code>:</p>
  
  <pre><code>Exception in thread "main" java.lang.ClassNotFoundException: org.apache.hadoop.fs.s3a.S3AFileSystem
      at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
      at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
      at java.lang.Class.forName0(Native Method)
      at java.lang.Class.forName(Class.java:348)
      at org.apache.spark.util.Utils$.classForName(Utils.scala:225)
      at org.apache.spark.deploy.history.HistoryServer$.main(HistoryServer.scala:256)
      at org.apache.spark.deploy.history.HistoryServer.main(HistoryServer.scala)
  </code></pre>
  
  <p>The Spark source code for loading configuration is different in <a href="https://github.com/apache/spark/blob/39e2bad6a866d27c3ca594d15e574a1da3ee84cc/core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala#L82-L125" rel="noreferrer">SparkSubmitArguments.scala</a>
  and in <a href="https://github.com/apache/spark/blob/39e2bad6a866d27c3ca594d15e574a1da3ee84cc/core/src/main/scala/org/apache/spark/deploy/history/HistoryServerArguments.scala#L67" rel="noreferrer">HistoryServerArguments.scala</a>, in particular the HistoryServerArguments does not seem to load packages.</p>
  
  <p>Is there a way to add the <code>org.apache.hadoop.fs.s3a.S3AFileSystem</code> dependency to the History Server?</p>