title: Running a python Apache Beam Pipeline on Spark
body: <p>I am giving apache beam (with python sdk) a try here so I created a simple pipeline and I tried to deploy it on a Spark cluster.</p>
  <pre class="lang-py prettyprint-override"><code>from apache_beam.options.pipeline_options import PipelineOptions
  import apache_beam as beam
  
  op = PipelineOptions([
          &quot;--runner=DirectRunner&quot;
      ]
  )
  
  
  with beam.Pipeline(options=op) as p:
      p | beam.Create([1, 2, 3]) | beam.Map(lambda x: x+1) | beam.Map(print)
  </code></pre>
  <p>This pipeline is working well with DirectRunner. So to deploy the same code on Spark (as the portability is a key concept in Beam)...</p>
  <p>First I edited the <code>PipelineOptions</code> as mentioned <a href="https://beam.apache.org/documentation/runners/spark/" rel="nofollow noreferrer">here</a>:</p>
  <pre class="lang-py prettyprint-override"><code>op = PipelineOptions([
          &quot;--runner=PortableRunner&quot;,
          &quot;--job_endpoint=localhost:8099&quot;,
          &quot;--environment_type=LOOPBACK&quot;
      ]
  )
  </code></pre>
  <p><code>job_endpoint</code> is the url to the docker container of the <a href="https://hub.docker.com/r/apache/beam_spark_job_server/tags" rel="nofollow noreferrer">beam spark job server</a> that I run using the command:</p>
  <pre class="lang-sh prettyprint-override"><code>docker run --net=host apache/beam_spark_job_server:latest --spark-master-url=spark://SPARK_URL:SPARK_PORT
  </code></pre>
  <p>This is supposed to work well but the job fails on Spark with this error :</p>
  <pre><code>20/10/31 14:35:58 ERROR TransportRequestHandler: Error while invoking RpcHandler#receive() for one-way message.
  
  java.io.InvalidClassException: org.apache.spark.deploy.ApplicationDescription; local class incompatible: stream classdesc serialVersionUID = 6543101073799644159, local class serialVersionUID = 1574364215946805297
  </code></pre>
  <p>Also, I have this WARN in the <code>beam_spark_job_server</code> logs:</p>
  <pre><code>WARN org.apache.beam.runners.spark.translation.SparkContextFactory: Creating a new Spark Context.
  </code></pre>
  <p>Any idea where is the problem here? Is there any other way to run python Beam Pipelines on spark without passing by a containerized service ?</p>