title: ignoreCorruptFiles option does not ignore corrupt files when one file having more than X records
body: <p>I'm trying to read a parquet folder having some files with different schema. I tried to use the option ignoreCorruptFiles=True to ignore the files having schema changes compare to the first file. But it does not work at all. Could you please give me an advice?</p>
  <p>Precondition:</p>
  <p>Spark 3.1 run locally on my Macbook Pro(16G Ram,i7, 2015)</p>
  <p>In folder A having two parquet files</p>
  <p>File 1: have some columns and one of them is column C1 with data type Int and have only one record</p>
  <p>File 2: Same schema with File 1 except column C1  having data type String and having&gt;= X records</p>
  <p>X depends on the capacity of your computer, my case is 36, you can increase the number of row to find X.</p>
  <p>Read file 1 to get schema of file 1.</p>
  <p>Read folder A with schema of file 1.</p>
  <p>Expected: Read successfully, file 2 will be ignored as the data type of column C1 changed to string.</p>
  <p>Actual: File 2 seems to be not ignored and get error:</p>
  <p> <code>WARN TaskSetManager: Lost task 0.0 in stage 2.0 (TID 2) (192.168.1.78 executor driver): java.lang.UnsupportedOperationException: org.apache.parquet.column.values.dictionary.PlainValuesDictionary$PlainBinaryDictionary WARN TaskSetManager: Lost task 0.0 in stage 2.0 (TID 2) (192.168.1.78 executor driver): java.lang.UnsupportedOperationException: org.apache.parquet.column.values.dictionary.PlainValuesDictionary$PlainBinaryDictionary at org.apache.parquet.column.Dictionary.decodeToInt(Dictionary.java:45)</code></p>
  <p> </p>
  <p>If i remove one record from file2. It works well  </p>
  <p>Code with newly creating files</p>
  <pre><code>spark.conf.set('spark.sql.files.ignoreCorruptFiles', True)
  schema1 = StructType([
   StructField(&quot;program_sk&quot;, IntegerType(), True),
   StructField(&quot;client_sk&quot;, IntegerType(), True),
  ])
  
  sample_data = [(1, 17)]
  df1 = spark.createDataFrame(sample_data, schema1)
  
  schema2 = StructType([
   StructField(&quot;program_sk&quot;, IntegerType(), True),
   StructField(&quot;client_sk&quot;, StringType(), True),
  ])
  sample_data = [(1, &quot;19999&quot;), (2, &quot;3332&quot;), (3, &quot;199999&quot;), (4, &quot;3333&quot;),
   (2, &quot;19999&quot;), (2, &quot;3332&quot;), (3, &quot;199999&quot;), (4, &quot;3333&quot;),
   (3, &quot;19999&quot;), (2, &quot;3332&quot;), (3, &quot;199999&quot;), (4, &quot;3333&quot;),
   (4, &quot;19999&quot;), (2, &quot;3332&quot;), (3, &quot;199999&quot;), (4, &quot;3333&quot;),
   (5, &quot;19999&quot;), (2, &quot;3332&quot;), (3, &quot;199999&quot;), (4, &quot;3333&quot;),
   (6, &quot;19999&quot;), (2, &quot;3332&quot;), (3, &quot;199999&quot;), (4, &quot;3333&quot;),
   (7, &quot;19999&quot;), (2, &quot;3332&quot;), (3, &quot;199999&quot;), (4, &quot;3333&quot;),
   (8, &quot;19999&quot;), (2, &quot;3332&quot;), (3, &quot;199999&quot;), (4, &quot;3333&quot;),
   (9, &quot;19999&quot;), (2, &quot;3332&quot;), (3, &quot;199999&quot;), (4, &quot;3333&quot;),
   ]
  df2 = spark.createDataFrame(sample_data, schema2)
  file_save_path = 's3://xxx-data-dev/adp_data_lake/test_ignore_corrupt/'
  
  df1.write \
   .mode('overwrite') \
   .format('parquet') \
   .save(f'{file_save_path}')
  
  df2.write \
   .mode('append') \
   .format('parquet') \
   .save(f'{file_save_path}')
  
  df = spark.read.schema(schema1).parquet(file_save_path)
  df.show()
  </code></pre>
  <p>I also created an issue in the Spark Jira Board
  <a href="https://issues.apache.org/jira/browse/SPARK-36983" rel="nofollow noreferrer">https://issues.apache.org/jira/browse/SPARK-36983</a></p>