title: Read CSV files having different schema in Spark (Pyspark) Streaming
body: <p>Let's say I have spark streaming job with csv files in a directory and I have defined schemaInference as true :</p>
  <pre><code>spark.sql(&quot;set spark.sql.streaming.schemaInference=true&quot;)
  input_df = spark.readStream.format(&quot;csv&quot;).option(&quot;inferSchema&quot;, &quot;true&quot;).option(&quot;header&quot;, &quot;true&quot;).option(&quot;delimiter&quot;, &quot;,&quot;).option(&quot;maxFilesPerTrigger&quot;, 1).csv(&quot;inputs/&quot;)
  </code></pre>
  <p>When I upload the first file in my input directory with column as <strong>ID, Salary</strong>, the spark job will infer the schema and read the file and write to the sink. But can I upload another file <strong>(with different schema)</strong> having columns as <strong>ID, manager</strong> in same directory and spark read it and write to the sink? I don't mind if the writestream is of format json file.</p>