Spark, Scala, Java, Big Data, Data Science.
the Spark logs are appearing as errors. This is because Spark, by default, outputs its logs to SYSTEM_ERR. Additionally, there is an extensive amount of Spark log. This can be limited by configuring Spark to display only messages above the WARN level. Spark 3.5.0 uses log4j2 for logging. Therefore,