title: Separate logs from Apache spark
body: <p>I would like to have separate log files from workers, masters and jobs(executors, submits, don't know how call it). I tried configuration in <code>log4j.properties</code> like </p>
  
  <pre><code>log4j.appender.myAppender.File=/some/log/dir/${log4j.myAppender.FileName}
  </code></pre>
  
  <p>and than passing <code>log4j.myAppender.FileName</code> in <code>SPARK_MASTER_OPTS</code>, <code>SPARK_WORKER_OPTS</code>, <code>spark.executor.extraJavaOptions</code> and <code>spark.driver.extraJavaOptions</code>. </p>
  
  <p>It works perfectly well with workers and masters but fails with executors and drivers. Here is example of how I use these:</p>
  
  <pre><code>./spark-submit ... --conf "\"spark.executor.extraJavaOptions=log4j.myAppender.FileName=myFileName some.other.option=foo\"" ...
  </code></pre>
  
  <p>I also tried putting <code>log4j.myAppender.FileName</code> with some default value in <code>spark-defaults.conf</code> but it doesn't work neither.</p>
  
  <p>Is there some way to achieve what I want?</p>