title: Airflow ModuleNotFoundError: No module named 'pyspark'
body: <p>I installed Airflow on my machine which works well and I have a local spark also (which is operational too).
  I want to use airflow to orchestrate two sparks tasks: <code>task_spark_datatransform &gt;&gt; task_spark_model_reco</code>.
  The two pyspark modules associated to these two tasks are tested and work well under spark. </p>
  
  <p>I also create a very simple Airflow Dag using <code>bashOperator</code>* to run each spark task. For example, for the task <code>task_spark_datatransform</code> I have: </p>
  
  <pre><code>task_spark_datatransform = BashOperator (task_id = 'task_spark_datatransform', bash_command = spark_home + 'spark-submit --master local [*]' + srcDir + 'dataprep.py'),
  where, in my case, spark_home = '/usr/bin/spark/bin/'
  </code></pre>
  
  <p>*As indicated in several serious tutorials on this same subject.</p>
  
  <p>Question: Why doesn't Airflow recognize pyspark?</p>
  
  <p>Log:</p>
  
  <pre><code>[2019-09-20 10:21:21 +0200] [5945] [INFO] Worker exiting (pid: 5945)
  [2019-09-20 10:21:51 +0200] [5554] [INFO] Handling signal: ttin
  [2019-09-20 10:21:51 +0200] [6128] [INFO] Booting worker with pid: 6128
  [2019-09-20 10:21:51,609] {__init__.py:51} INFO - Using executor SequentialExecutor
  [2019-09-20 10:21:52,021] {__init__.py:305} INFO - Filling up the DagBag from /home/ach/airflow/dags
  [2019-09-20 10:21:52,026] {__init__.py:416} ERROR - Failed to import: /home/ach/airflow/dags/spark_af.py
  Traceback (most recent call last):
    File "/home/ach/airflow/lib/python3.7/site-packages/airflow/models/__init__.py", line 413, in process_file
      m = imp.load_source(mod_name, filepath)
    File "/home/ach/airflow/lib/python3.7/imp.py", line 171, in load_source
      module = _load(spec)
    File "&lt;frozen importlib._bootstrap&gt;", line 696, in _load
    File "&lt;frozen importlib._bootstrap&gt;", line 677, in _load_unlocked
    File "&lt;frozen importlib._bootstrap_external&gt;", line 728, in exec_module
    File "&lt;frozen importlib._bootstrap&gt;", line 219, in _call_with_frames_removed
    File "/home/ach/airflow/dags/spark_af.py", line 3, in &lt;module&gt;
      import dataprep
    File "/home/ach/airflow/dags/dataprep.py", line 2, in &lt;module&gt;
      from pyspark.sql import SparkSession
  ModuleNotFoundError: No module named 'pyspark'
  </code></pre>