title: best practice for debugging python-spark code
body: <p>I want to step through a python-spark code while still using yarn. The way I current do it is to start pyspark shell, copy-paste and then execute the code line by line.  I wonder whether there is a better way. </p>
  
  <p><code>pdb.set_trace()</code> would be a much more efficient option if it works. I tried it with <code>spark-submit --master yarn --deploy-mode client</code>. The program did stop and give me a shell at the line where <code>pdb.set_trace()</code> was called. However, any pdb commands entered in the shell simply hanged.  The <code>pdb.set_trace()</code> was inserted between spark function calls which, as I understand, should be executed in the driver that runs locally and with a terminal attached. 
  I read this post <a href="https://stackoverflow.com/questions/31245083/how-can-pyspark-be-called-in-debug-mode">How can pyspark be called in debug mode?</a> which seems to suggest the use of pdb is impossible without relying IDE(PyCharm). However, if interactively running spark code is possible, there should be a way to ask python-spark "run all the way until this line and give me a shell for REPL(interactive use). I haven't found any ways to do this.   Any suggestions/references are appreciated. </p>