title: How to use incremental data to create dataframes in pyspark
body: <p>I have some tables in hive. These tables get data appended incrementally to them.</p>
  
  <p>Now I have created a <code>data frame</code> in pyspark using a table in hive today. I have done a transpose on the <code>data frame</code> and created another table with the new transposed <code>data frame</code> in hive.</p>
  
  <p>Say tomorrow I get new incremental data in hive table of 100  new rows. Now I want to use only these 100 new rows an create a new <code>data frame</code> and do a transpose and append to the existing transposed hive table.</p>
  
  <p>How can I achieve that using pyspark.  </p>