Question: I run Spark in standalone mode. It reads a few files as `RDD`, process them, union the results in a single data frame and save the data frame to a parquet file in a local disk.

```scala
df.write.mode(SaveMode.Overwrite).parquet(output)

```

This process generates a noisy output. I have 30Gb of logs (my output parquet file is 10 times less!), mostly in this like format: 

```scala
root
 |-- column1: string (nullable = true)
 |-- column2: string (nullable = true)
 |-- column3: string (nullable = true)
 |-- column4: string (nullable = true)
 |-- column5: string (nullable = true)

```

How can I suppress the output?

I have already tried standard suppression tricks, none of them helped.

```scala
sc.setLogLevel("FATAL")

```

and

```scala
Logger.getLogger("org").setLevel(Level.OFF)
Logger.getLogger("akka").setLevel(Level.OFF)

```

Answer: 
Add a `log4j.properties` in the root directory of the jar used in `spark-submit`. Or if you are using `intellij` put it in `src/main/resources`

Place the following inside the `log4j.properties`

```
# Set everything to be logged to the file bagel/target/unit-tests.log
log4j.rootCategory=WARN, console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n

# Ignore messages below warning level from Jetty, because it's a bit verbose
# Settings to quiet third party logs that are too verbose
log4j.logger.org.eclipse.jetty=WARN
log4j.logger.org.eclipse.jetty.util.component.AbstractLifeCycle=ERROR
log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO
log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO
log4j.logger.org.apache.spark=WARN
log4j.logger.parquet=ERROR
log4j.logger.org.apache.spark.sql.execution.datasources.parquet=ERROR
log4j.logger.org.apache.spark.sql.execution.datasources.FileScanRDD=ERROR
log4j.logger.org.apache.hadoop.io.compress.CodecPool=ERROR

```

**Update** An even more solid way to suppress logging:

Add the above `log4j.properties` into the `$SPARK_HOME/conf` dir.
I use the following method :

```
  def suppressParquetLogging(): Unit = {
    // Based on https://issues.apache.org/jira/browse/SPARK-8118
    val contents =
      """
        |org.apache.parquet.handlers=java.util.logging.ConsoleHandler
        |java.util.logging.ConsoleHandler.level=SEVERE
      """.stripMargin
    val inputStream = new ByteArrayInputStream(contents.getBytes(StandardCharsets.UTF_8))
    java.util.logging.LogManager.getLogManager.readConfiguration(inputStream)
  }

```

It assumes you don't normally use java.util.logging.LogManager for logging