title: Spark Avro to parquet writing null values in number fields
body: <p>I am saving a spark dataframe as a parquet file and the dataframe has rows built from an avro object. The exact code for the same is here - <a href="https://stackoverflow.com/a/41491999/2440775">https://stackoverflow.com/a/41491999/2440775</a> </p>
  
  <p>The challenge I am facing is, I intend to be able to have nulls when the integer fields are missing in the incoming data. Avro seems to allow that by using Union types, but when I do not specify default value or specify a default value of "null" in the avsc, I get errors like:</p>
  
  <pre><code>Caused by: org.apache.avro.AvroRuntimeException: Field xxx type:LONG pos:7 not set and has no default value
      at org.apache.avro.generic.GenericData.getDefaultValue(GenericData.java:984)
      at org.apache.avro.data.RecordBuilderBase.defaultValue(RecordBuilderBase.java:135)
  
  Or
  
  Caused by: org.apache.avro.AvroRuntimeException: Field xxx type:UNION pos:7 not set and has no default value
      at org.apache.avro.generic.GenericData.getDefaultValue(GenericData.java:984)
      at org.apache.avro.data.RecordBuilderBase.defaultValue(RecordBuilderBase.java:135)
  </code></pre>
  
  <p>If I write a default value of "0", the it saveAsParquet works well</p>
  
  <p>I also tried changing the avro spec to have the "null" type first since the union picks the type of first element. </p>
  
  <pre><code>"type": ["null","long"], "default": null
  </code></pre>
  
  <p>This causes an exception as below:</p>
  
  <pre><code>org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0 (TID 0, localhost): java.lang.ClassCastException: java.lang.String cannot be cast to java.lang.Long
  </code></pre>
  
  <p>Changing the order of long and null in avro schema leads to the below exception</p>
  
  <p>Caused by: org.apache.avro.AvroTypeException: Non-numeric default value for long: null</p>