title: Merge two spark dataframes based on a column
body: <p>I have 2 dataframes which I need to merge based on a column (Employee code).  Please note that the dataframe has about 75 columns, so I am providing a sample dataset to get some suggestions/sample solutions.  I am using databricks, and the datasets are read from S3.</p>
  
  <p>Following are my 2 dataframes:</p>
  
  <pre><code>DATAFRAME - 1
  
  |-----------------------------------------------------------------------------------|
  |EMP_CODE   |COLUMN1|COLUMN2|COLUMN3|COLUMN4|COLUMN5|COLUMN6|COLUMN7|COLUMN8|COLUMN9|
  |-----------------------------------------------------------------------------------|
  |A10001     |   B   |       |       |       |       |       |       |       |       |
  |-----------------------------------------------------------------------------------|
  
  
  DATAFRAME - 2
  |-----------------------------------------------------------------------------------|
  |EMP_CODE   |COLUMN1|COLUMN2|COLUMN3|COLUMN4|COLUMN5|COLUMN6|COLUMN7|COLUMN8|COLUMN9|
  |-----------------------------------------------------------------------------------|
  |A10001     |       |       |       |       |   C   |       |       |       |       |   
  |B10001     |       |       |       |       |       |       |       |       |T2     |
  |A10001     |       |       |       |       |       |       |       |   B   |       |
  |A10001     |       |       |   C   |       |       |       |       |       |       |
  |C10001     |       |       |       |       |       |   C   |       |       |       |
  |-----------------------------------------------------------------------------------|   
  </code></pre>
  
  <p>I need to merge the 2 dataframes based on EMP_CODE, basically join dataframe1 with dataframe2, based on emp_code.  I am getting duplicate columns when i do a join, and I am looking for some help.</p>
  
  <p>Expected final dataframe:</p>
  
  <pre><code>|-----------------------------------------------------------------------------------|
  |EMP_CODE   |COLUMN1|COLUMN2|COLUMN3|COLUMN4|COLUMN5|COLUMN6|COLUMN7|COLUMN8|COLUMN9|
  |-----------------------------------------------------------------------------------|
  |A10001     |   B   |       |   C   |       |   C   |       |       |   B   |       |   
  |B10001     |       |       |       |       |       |       |       |       |T2     |
  |C10001     |       |       |       |       |       |   C   |       |       |       |
  |-----------------------------------------------------------------------------------|       
  </code></pre>
  
  <p>There are 3 rows with emp_code A10001 in dataframe1, and 1 row in dataframe2.  All data should be merged as one record without any duplicate columns.</p>
  
  <p>Thanks much</p>