title: pySpark how does TempView table Joined to Hive table
body: <p>I have a dataFrame which is registered as tempView and a Hive table to join</p>
  
  <pre><code>    df1.createOrReplaceTempView("mydata")
  
      df2 = spark.sql("Select md.column1,md.column2,mht.column1 \
                      from mydata md inner join myHivetable mht on mht.key1 = md.key1 \
                       where mht.transdate between '2017-08-01' and '2017-08-10' ")
  </code></pre>
  
  <p>How does this join happen. Will spark try to read the hive table into the memory or decide to write the tempView  table into hive if the data volume is very high in the Hive table.</p>
  
  <p><strong><em>Adding the following after the first answer for additional details:</em></strong></p>
  
  <p>Let's say we have </p>
  
  <p>100 row as tempView in Spark called TABLE_A.</p>
  
  <p>A 1 Billion row table in Hive TABLE_B. </p>
  
  <p><strong>Next step we need to join TABLE_A with TABLE_B .</strong>  </p>
  
  <p><em>There is a date range condition on TABLE_B.</em> </p>
  
  <p>Since  table TABLE_B is big in size. Will spark read the whole table TABLE_B into memory or decide to write TABLE_A to temp space in Hadoop to do a Hive Join or how intelligent it will figure out the best way to do the join for performance</p>