title: What is the right way to pass external libraries to spark-submit command line?
body: <p>I am currently trying to submit a spark application that depends on several third party libraries (apache tika, ...), and I want to use a custom log4j configuration for my executors.
  I browsed dozens of questions here is stackoverflow as well as spark official documentation without figuring out the exact behavior I could expect, so I will try to sum up here the context and what I don't understand:
  First, to explain the context : I use spark 2.3.0 on yarn, in cluster mode (HDP platform).</p>
  <p>Basically, I first tried to launch something like :</p>
  <pre><code>spark-submit --master yarn --deploy-mode=cluster --files ./log4j.properties,./myConfig.properties --class org.company.MyClass --conf 'spark.executor.extraJavaOptions=-Dlog4j.configuration=log4j.properties' ./target/myApplicationUber.jar
  </code></pre>
  <p>First issue :
  Like many of you, I faced issues because of dependencies conflicts : although I provided a uber jar file that contained all the classes from the dependencies, some of those classes were already provided by hadoop or spark libraries (lower versions) and it caused troubles to my executors at runtime</p>
  <p>As a workaround, I thought I had to prepend my executors classpath with the right versions of those dependencies, so I tried out following:
  I added the '--files' option to specifically distributes the libraries jar files to the executors, but realized it was not enough, because they were not added to the classpath, so finally tried following command line:</p>
  <pre><code>spark-submit --master yarn --deploy-mode=cluster --files ./log4j.properties,./myConfig.propertie,/absolute/path/to/tika-app.jar,/absolute/path/to/lib2.jar --class org.company.MyClass --conf spark.executor.extraClassPath=tika-app.jar:lib2.jar --conf 'spark.executor.extraJavaOptions=-Dlog4j.configuration=log4j.properties' ./target/myApplicationUber.jar
  </code></pre>
  <p>It worked pretty well, except that I realized that tika-app.jar also includes a &quot;log4j.properties&quot; configuration file and that it was used instead of mine...</p>
  <p><strong>I couldn't figure out why the file I was explicitely providing both with the --files option and inside my application uber jar was not taking precedence over the other one ?</strong></p>
  <p>I finally had to rename my file to myLog4j.properties and modify my command line to make it working:</p>
  <pre><code>spark-submit --master yarn --deploy-mode=cluster --files ./myLog4j.properties,./myConfig.propertie,/absolute/path/to/tika-app.jar,/absolute/path/to/lib2.jar --class org.company.MyClass --conf spark.executor.extraClassPath=tika-app.jar:lib2.jar --conf 'spark.executor.extraJavaOptions=-Dlog4j.configuration=myLog4j.properties' ./target/myApplicationUber.jar
  </code></pre>
  <p><strong>Additional question:</strong></p>
  <p>More recently, I realized I could use the --jars option to distribute dependencies jar files (instead of --files) : spark documentation states that <em>&quot;...That list is included in the driver and executor classpaths...&quot;</em>, so I thought it was not necessary anymore to pass the spark.executor.extraClassPath option, and gave a try to following command line :</p>
  <pre><code>spark-submit --master yarn --deploy-mode=cluster --files ./myLog4j.properties,./myConfig.properties --jars /absolute/path/to/tika-app.jar,/absolute/path/to/lib2.jar --class org.company.MyClass --conf 'spark.executor.extraJavaOptions=-Dlog4j.configuration=myLog4j.properties' ./target/myApplicationUber.jar
  </code></pre>
  <p>But I came back to first dependencies issue (class versions), so I guess tika-app.jar and lib2.jar have not been added to my executors classpath...</p>
  <p><strong>Is there a problem in the documentation or am I wrong somewhere else ?</strong></p>
  <p><strong>Final question:</strong></p>
  <p>while struggling with classpath issues, I wondered several times what classloader strategy was involved between spark itself and my application jar file ? When the JVM starts in the executor, why isn't there isolation between all the classes I provided in the uber jar and &quot;the rest of the world&quot; ? Wouldn't it make easier to avoid conflicts between different versions of a same library ?</p>
  <p>Thanks to those who were patient enough to read till the end :)</p>