title: How to overwrite entire existing column in Spark dataframe with new column?
body: <p>I want to overwrite a spark column with a new column which is a binary flag.</p>
  
  <p>I tried directly overwriting the column id2 but why is it not working like a inplace operation in Pandas?</p>
  
  <p>How to do it without using withcolumn() to create new column and drop() to drop the old column?</p>
  
  <p>I know that spark dataframe is immutable, is that the reason or there is a different way to overwrite without using withcolumn() &amp; drop()?</p>
  
  <pre><code>    df2 = spark.createDataFrame(
          [(1, 1, float('nan')), (1, 2, float(5)), (1, 3, float('nan')), (1, 4, float('nan')), (1, 5, float(10)), (1, 6, float('nan')), (1, 6, float('nan'))],
          ('session', "timestamp1", "id2"))
  
      df2.select(df2.id2 &gt; 0).show()
  
  +---------+
  |(id2 &gt; 0)|
  +---------+
  |     true|
  |     true|
  |     true|
  |     true|
  |     true|
  |     true|
  |     true|
  +---------+
   # Attempting to overwriting df2.id2
      df2.id2=df2.select(df2.id2 &gt; 0).withColumnRenamed('(id2 &gt; 0)','id2')
      df2.show()
  #Overwriting unsucessful
  +-------+----------+----+
  |session|timestamp1| id2|
  +-------+----------+----+
  |      1|         1| NaN|
  |      1|         2| 5.0|
  |      1|         3| NaN|
  |      1|         4| NaN|
  |      1|         5|10.0|
  |      1|         6| NaN|
  |      1|         6| NaN|
  +-------+----------+----+
  </code></pre>