title: why spark to_json() not populating null values?
body: <p>Can try in spark-shell</p>
  
  <pre><code>case class Employee(id: Int, name: String, department: String, salary: Option[Double])
  import org.apache.spark.sql.functions._
  import spark.implicits._
  
  case class Employee(id: Int, name: String, department: String, salary: Option[Double])
  val data = List(Employee(1, "XYZ", "dep1", Some(1234.0)), Employee(0, null, "unknown", None)).toDS()
  data.select($"id", to_json(struct($"id",$"name", $"department", $"salary")).as("json_data")).show(false)
  </code></pre>
  
  <p>return =></p>
  
  <pre><code>|id |json_data                                                |
  +---+---------------------------------------------------------+
  |1  |{"id":1,"name":"XYZ","department":"dep1","salary":1234.0}|
  |0  |{"id":0,"department":"unknown"}                          |
  </code></pre>
  
  <p>expecting =></p>
  
  <pre><code>|id |json_data                                                   |
  +---+------------------------------------------------------------+
  |1  |{"id":1,"name":"XYZ","department":"dep1","salary":1234.0}   |
  |0  |{"id":0,"name": null, "department":"unknown","salary":null} |
  </code></pre>
  
  <p>null fields(<strong>name &amp; salary</strong>) also should be populated in resulting json. I don't want to use lit("null") to populate null values</p>