title: Insert Overwrite in data bricks overwriting complete data in table?
body: <p>I am have two table 1 is with 50K records and other is with 2.5K records and I want to update this 2.5K records into table one. Currently I was doing this by using INSERT OVERWRITE statement in spark Mapr cluster. And I want to do same in azure databricks. Where I created 2 tables in databricks and read data from on-prem servers into azure then using INSERT OVERWRITE statement. But I was doing this my previus/History data was completely replaced with new data.</p>
  <p><strong>In Mapr cluster</strong></p>
  <pre><code>src_df_name.write.mode(&quot;overwrite&quot;).format(&quot;hive&quot;).saveAsTable(s&quot;cs_hen_mbr_stg&quot;) //stage table with 2.5K records.
  
  spark.sql(s&quot;INSERT OVERWRITE TABLE cs_hen_mbr_hist &quot; +
            s&quot;SELECT NAMED_STRUCT('INID',stg.INID,'SEG_NBR',stg.SEG_NBR,'SRC_ID',stg.SRC_ID, &quot;+
            s&quot;'SYS_RULE',stg.SYS_RULE,'MSG_ID',stg.MSG_ID, &quot; +
            s&quot;'TRE_KEY',stg.TRE_KEY,'PRO_KEY',stg.PRO_KEY, &quot; +
            s&quot;'INS_DATE',stg.INS_DATE,'UPDATE_DATE',stg.UPDATE_DATE,'STATUS_KEY',stg.STATUS_KEY) AS key, &quot;+
            s&quot;stg.MEM_KEY,stg.INDV_ID,stg.MBR_ID,stg.SEGO_ID,stg.EMAIL, &quot; +
            s&quot;from cs_hen_mbr_stg stg&quot; )
  </code></pre>
  <p>By doing above in mapr cluster I was able to update values.But i was trying same in azure databricks My history data is getting lost.</p>
  <p><strong>In Databriks</strong></p>
  <pre><code>val VW_HISTORY_MAIN=spark.read.format(&quot;parquet&quot;).option(&quot;header&quot;,&quot;true&quot;).load(s&quot;${SourcePath}/VW_HISTORY&quot;)
  VW_HISTORY_MAIN.write.mode(&quot;overwrite&quot;).format(&quot;hive&quot;).saveAsTable(s&quot;demo.cs_hen_mbr_stg&quot;) //writing this to table in databricks.
  
  spark.sql(s&quot;INSERT OVERWRITE TABLE cs_hen_mbr_hist &quot; +
            s&quot;SELECT NAMED_STRUCT('INID',stg.INID,'SEG_NBR',stg.SEG_NBR,'SRC_ID',stg.SRC_ID, &quot;+
            s&quot;'SYS_RULE',stg.SYS_RULE,'MSG_ID',stg.MSG_ID, &quot; +
            s&quot;'TRE_KEY',stg.TRE_KEY,'PRO_KEY',stg.PRO_KEY, &quot; +
            s&quot;'INS_DATE',stg.INS_DATE,'UPDATE_DATE',stg.UPDATE_DATE,'STATUS_KEY',stg.STATUS_KEY) AS key, &quot;+
            s&quot;stg.MEM_KEY,stg.INDV_ID,stg.MBR_ID,stg.SEGO_ID,stg.EMAIL, &quot; +
            s&quot;from cs_hen_mbr_stg stg&quot; )
  
  </code></pre>
  <p>Why it is not working with databricks?</p>