title: Read Zstandard-compressed file in Spark 2.3.0
body: <p>Apache Spark supposedly supports Facebook's Zstandard compression algorithm as of Spark 2.3.0 (<a href="https://issues.apache.org/jira/browse/SPARK-19112" rel="noreferrer">https://issues.apache.org/jira/browse/SPARK-19112</a>), but I am unable to actually read a Zstandard-compressed file:</p>
  
  <pre><code>$ spark-shell
  
  ...
  
  // Short name throws an exception
  scala&gt; val events = spark.read.option("compression", "zstd").json("data.zst")
  java.lang.IllegalArgumentException: Codec [zstd] is not available. Known codecs are bzip2, deflate, uncompressed, lz4, gzip, snappy, none.
  
  // Codec class can be imported
  scala&gt; import org.apache.spark.io.ZStdCompressionCodec
  import org.apache.spark.io.ZStdCompressionCodec
  
  // Fully-qualified code class bypasses error, but results in corrupt records
  scala&gt; spark.read.option("compression", "org.apache.spark.io.ZStdCompressionCodec").json("data.zst")
  res4: org.apache.spark.sql.DataFrame = [_corrupt_record: string]
  </code></pre>
  
  <p>What do I need to do in order to read such a file?</p>
  
  <p>Environment is AWS EMR 5.14.0.</p>