title: Reading csv files with missing columns and random column order
body: <p>I have a schema that I want to apply to csv files in Databricks.
  The csv files may contain 6 columns (a,b,c,d,e,f) which can appear in random order in the csv files. It can also occur that one or more columns are missing. So csv files with these headers would be valid</p>
  <pre><code>a,b,c,d,e,f
  f,e,d,c,a,b
  a,b,c
  d,e,f
  </code></pre>
  <p>I can create a custom schema, but this does not handle the different order, as well as missing columns. They are applied sequential. Any ideas on how this can be dealt with?</p>
  <pre class="lang-py prettyprint-override"><code>customSchema = StructType() \
    .add(&quot;a&quot;, DoubleType(), True) \
    .add(&quot;b&quot;, DoubleType(), True) \
    .add(&quot;c&quot;, DoubleType(), True) \
    .add(&quot;d&quot;, DoubleType(), True) \
    .add(&quot;e&quot;, DoubleType(), True) \
    .add(&quot;f&quot;, DoubleType(), False)
  
   
  data = sqlContext.read.format(&quot;csv&quot;) \
    .option(&quot;header&quot;, &quot;true&quot;) \
    .option(&quot;delimiter&quot;, &quot;,&quot;) \
    .schema(customSchema) \
    .load(&quot;*.csv&quot;)
  </code></pre>