title: Complex in memory object to Parquet files without Apache Spark
body: <p>I have an object that I would like to create a parquet file for each of those lists.
  That object looks like this:</p>
  
  <pre><code>case class ProgramsData(programs: List[Program], switches: List[Switch], paths: List[Path],
                      activities: List[Activity], enactments: List[Enactment])
  </code></pre>
  
  <p>After some research, I'm a bit baffled as to how to achieve this.</p>
  
  <p>It seems that the only way to do this without Apache Spark is to convert your object into an Avro file, then read that in and get the schema, to which you can then create a parquet file from.</p>
  
  <p>Is there a way to cut the middle man out and convert my object to a parquet file for each list inside the object?
  If not, what's the simplest way to achieve my end goal? 
  Sadly most of the few examples I've seen either don't work or don't exactly do what I want.</p>
  
  <p>Thanks in advance.</p>
  
  <p><em>Edit</em>
  So after another full day working on this, I've produced some code that generates a file. But when writing some code to read in the parquet files, it tells me it cannot decode it. Which tells me the writer code is wrong (though surely if it can't create a valid parquet file, it would throw an exception?)</p>
  
  <p>Here is the writing code:</p>
  
  <pre><code>val avroSchema: Schema = ReflectData.get().getSchema(classOf[Program])
  
    val parquetOutputPath = new org.apache.hadoop.fs.Path(outputFilePath)
  
    val parquetWriter = new AvroParquetWriter[Record](parquetOutputPath, avroSchema)
  
    programs.foreach(program =&gt; {
      val programRecord = new GenericData.Record(avroSchema)
      programRecord.put("name", program.getName)
  
      parquetWriter.write(programRecord)
    })
  
    parquetWriter.close()
  </code></pre>
  
  <p>And here is the reading code:</p>
  
  <pre><code> val avroSchema: Schema = ReflectData.get().getSchema(classOf[Program])
  
  val filePath = new org.apache.hadoop.fs.Path(filePathString)
  
  val parquetReader = new AvroParquetReader[Record](filePath)
  
  val record = parquetReader.read()
  </code></pre>
  
  <p>The last line of the reading code throws the following exception:</p>
  
  <blockquote>
    <p>.ParquetDecodingException: Can not read value at 1 in block 0 in file file:/tmp/parquet/test-1506962769.parquet</p>
  </blockquote>
  
  <p>Hopefully someone can point me in the right direction, otherwise I may have to use Apache Spark just for its ability to easily create a parquet file, which is overkill.</p>