title: Spark : multiple spark-submit in parallel
body: <p>I have a generic question about Apache Spark :</p>
  
  <p>We have some spark streaming scripts that consume Kafka messages.
  Problem : they are failing randomly without a specific error...</p>
  
  <p>Some script does nothing while they are working when I run them manually, one is failing with this message :</p>
  
  <blockquote>
    <p>ERROR SparkUI: Failed to bind SparkUI
    java.net.BindException: Address already in use: Service 'SparkUI' failed after 16 retries!</p>
  </blockquote>
  
  <p>So I'm wondering if there is maybe a specific way to run the scripts in parallel ?</p>
  
  <p>They are all in the same jar and I run them with Supervisor.
  Spark is installed on Cloudera Manager 5.4 on Yarn.</p>
  
  <p>Here is how I launch a script :</p>
  
  <pre><code>sudo -u spark spark-submit --class org.soprism.kafka.connector.reader.TwitterPostsMessageWriter /home/soprism/sparkmigration/data-migration-assembly-1.0.jar --master yarn-cluster --deploy-mode client
  </code></pre>
  
  <p>Thanks for your help !</p>
  
  <p>Update : I changed the command and now run this (it stops with now specific message) :</p>
  
  <pre><code>root@ns6512097:~# sudo -u spark spark-submit --class org.soprism.kafka.connector.reader.TwitterPostsMessageWriter --master yarn --deploy-mode client /home/soprism/sparkmigration/data-migration-assembly-1.0.jar
  SLF4J: Class path contains multiple SLF4J bindings.
  SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-5.4.7-1.cdh5.4.7.p0.3/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-5.4.7-1.cdh5.4.7.p0.3/jars/avro-tools-1.7.6-cdh5.4.7.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
  SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
  15/09/28 16:14:21 INFO Remoting: Starting remoting
  15/09/28 16:14:21 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@ns6512097.ip-37-187-69.eu:52748]
  15/09/28 16:14:21 INFO Remoting: Remoting now listens on addresses: [akka.tcp://sparkDriver@ns6512097.ip-37-187-69.eu:52748]
  SLF4J: Class path contains multiple SLF4J bindings.
  SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-5.4.7-1.cdh5.4.7.p0.3/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-5.4.7-1.cdh5.4.7.p0.3/jars/avro-tools-1.7.6-cdh5.4.7.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
  SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
  </code></pre>