I am trying to cast multiple date formats from string type field using Pyspark. When I am using below date format it is working fine.
def custom_to_date(col):
    formats = ("MM/dd/yyyy", "yyyy-MM-dd", "dd/MM/yyyy", "MM/yy","dd/M/yyyy")
    return coalesce(*[to_date(col, f) for f in formats])
    
from pyspark.sql.functions import coalesce, to_date
df = spark.createDataFrame([(1, "01/22/2010"), (2, "2018-12-01")], ("id", "dt"))
df.withColumn("pdt", custom_to_date("dt")).show()

Above code gives the correct output.
But when I use the month in single digit as below, the code fails.
df = spark.createDataFrame([(1, "01/22/2010"), (2, "2018-12-1"),(3,"24/7/2006")], ("id", "dt"))

I got the below error message.
org.apache.spark.SparkException:
  Job aborted due to stage failure:
    Task 2 in stage 2.0 failed 4 times, most recent failure: 
      Lost task 2.3 in stage 2.0 (TID 10) (10.13.82.55 executor 0): 
        org.apache.spark.SparkUpgradeException: 
        [INCONSISTENT_BEHAVIOR_CROSS_VERSION.PARSE_DATETIME_BY_NEW_PARSER] 
        You may get a different result due to the upgrading to Spark >= 3.0:


Spark is unable to handle a particular date format
Adding an answer since the comments and others answer doesn't cover the behaviour. The solution is not to add new formats. Since the formats itself can be better defined.
with spark 3.0 M supports 01, 1. January, Jan.
So you don't need MM
spark reference - https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html
def custom_to_date(col):
    formats = ("M/d/yyyy", "yyyy-M-d", "d/M/y", "M/y")
    return coalesce(*[to_date(col, f) for f in formats])
    
from pyspark.sql.functions import coalesce, to_date
df = spark.createDataFrame([(1, "01/22/2010"), (2, "2018-12-1"),(3,"12/2023")], ("id", "dt"))
df.withColumn("pdt", custom_to_date("dt")).show()

Results -

Alternatively, if you want legacy behavior then you can use
spark.conf.set("spark.sql.legacy.timeParserPolicy","LEGACY")
or
spark.sql("set spark.sql.legacy.timeParserPolicy=LEGACY")
