Question: I am using the following code to save a spark DataFrame to JSON file

```
unzipJSON.write.mode("append").json("/home/eranw/Workspace/JSON/output/unCompressedJson.json")

```

the output result is:

```
part-r-00000-704b5725-15ea-4705-b347-285a4b0e7fd8
.part-r-00000-704b5725-15ea-4705-b347-285a4b0e7fd8.crc
part-r-00001-704b5725-15ea-4705-b347-285a4b0e7fd8
.part-r-00001-704b5725-15ea-4705-b347-285a4b0e7fd8.crc
_SUCCESS
._SUCCESS.crc

```

1. How do I generate a single JSON file and not a file per line?
2. How can I avoid the \*crc files?
3. How can I avoid the SUCCESS file?

Answer: 
If you want a single file, you need to do a `coalesce` to a single partition before calling write, so:

```
unzipJSON.coalesce(1).write.mode("append").json("/home/eranw/Workspace/JSON/output/unCompressedJson.json")

```

Personally, I find it rather annoying that the number of output files depend on number of partitions you have before calling `write` - especially if you do a `write` with a `partitionBy` - but as far as I know, there are currently no other way.

I don't know if there is a way to disable the .crc files - I don't know of one - but you can disable the \_SUCCESS file by setting the following on the hadoop configuration of the Spark context.

```
sc.hadoopConfiguration.set("mapreduce.fileoutputcommitter.marksuccessfuljobs", "false")

```

Note, that you may also want to disable generation of the metadata files with:

```
sc.hadoopConfiguration.set("parquet.enable.summary-metadata", "false")

```

Apparently, generating the metadata files takes some time (see [this blog post](https://www.appsflyer.com/blog/the-bleeding-edge-spark-parquet-and-s3/)) but aren't actually that important (according to [this](http://search-hadoop.com/m/q3RTtrwSwe2hVzCu&subj=Re+Parquet+writing+gets+progressively+slower)). Personally, I always disable them and I have had no issues.
Just a little update in above answer. To disable crc and SUCCESS file, simply set property in spark session as follows(example)

```
spark = SparkSession \
        .builder \
        .appName("Python Spark SQL Hive integration example") \
        .config("spark.sql.warehouse.dir", warehouse_location) \
        .enableHiveSupport() \
        .getOrCreate()

    spark.conf.set("mapreduce.fileoutputcommitter.marksuccessfuljobs", "false")

```