title: Submit PySpark to Yarn cluster using Java
body: <p>i need to create a Java program that submit python scripts (that use PySpark) to a Yarn cluster.
  Now, i saw that using SparkLauncher is the same as using a YarnClient, because it uses a Yarn Client built-in (writing my own Yarn Client is insane, i tried, too much things to handle).
  So i wrote:</p>
  
  <pre><code> public static void main(String[] args) throws Exception {
      String SPARK_HOME = System.getProperty("SPARK_HOME");
      submit(SPARK_HOME, args);
  }
  
  static void submit(String SPARK_HOME, String[] args) throws Exception {
      String[] arguments = new String[]{
              // application name
              "--name",
              "SparkPi-Python",
  
              "--class",
              "org.apache.spark.deploy.PythonRunner",
  
              "--py-files",
              SPARK_HOME + "/python/lib/pyspark.zip,"+ SPARK_HOME +"/python/lib/py4j-0.9-src.zip",
  
              // Python Program
              "--primary-py-file",
              "/home/lorenzo/script.py",
  
              // number of executors
              "--num-executors",
              "2",
  
              // driver memory
              "--driver-memory",
              "512m",
  
              // executor memory
              "--executor-memory",
              "512m",
  
              // executor cores
              "--executor-cores",
              "2",
  
              "--queue",
              "default",
  
              // argument 1 to my Spark program
              "--arg",
              null,
      };
      System.setProperty("SPARK_YARN_MODE", "true");
      System.out.println(SPARK_HOME);
      SparkLauncher sparkLauncher = new SparkLauncher();
      sparkLauncher.setSparkHome("/usr/hdp/current/spark2-client");
      sparkLauncher.setAppResource("/home/lorenzo/script.py");
      sparkLauncher.setMaster("yarn");
      sparkLauncher.setDeployMode("cluster");
      sparkLauncher.setVerbose(true);
      sparkLauncher.launch().waitFor();
  }
  </code></pre>
  
  <p>When i run this Jar from a machine in the cluster, nothing happen... No Error, No Log, No yarn container... just nothing... if i try to put a println inside this code, obvs, it prints the println.</p>
  
  <ol>
  <li>What i'm misconfiguring? </li>
  <li>If i want run this JAR from a different machine, where and how should declare the IP?</li>
  </ol>