title: Apache Spark - Dataset operations fail in abstract base class?
body: <p>I'm trying to extract some common code into an abstract class, but running into issues.  </p>
  
  <p>Let's say I'm reading in a file formatted as "id|name":</p>
  
  <pre><code>case class Person(id: Int, name: String) extends Serializable
  
  object Persons {
    def apply(lines: Dataset[String]): Dataset[Person] = {
      import lines.sparkSession.implicits._
      lines.map(line =&gt; {
        val fields = line.split("\\|")
        Person(fields(0).toInt, fields(1))
      })
    }
  }
  
  Persons(spark.read.textFile("persons.txt")).show()
  </code></pre>
  
  <p>Great. This works fine.  Now let's say I want to read a number of different files with "name" fields, so I'll extract out all of the common logic:</p>
  
  <pre><code>trait Named extends Serializable { val name: String }
  
  abstract class NamedDataset[T &lt;: Named] {
    def createRecord(fields: Array[String]): T
    def apply(lines: Dataset[String]): Dataset[T] = {
      import lines.sparkSession.implicits._
      lines.map(line =&gt; createRecord(line.split("\\|")))
    }
  }
  
  case class Person(id: Int, name: String) extends Named
  
  object Persons extends NamedDataset[Person] {
    override def createRecord(fields: Array[String]) =
      Person(fields(0).toInt, fields(1))
  }
  </code></pre>
  
  <p>This fails with two errors:</p>
  
  <pre><code>Error:
  Unable to find encoder for type stored in a Dataset.  
  Primitive types (Int, String, etc) and Product types (case classes) 
  are supported by importing spark.implicits._  Support for serializing 
  other types will be added in future releases.
  lines.map(line =&gt; createRecord(line.split("\\|")))
  
  Error:
  not enough arguments for method map: 
  (implicit evidence$7: org.apache.spark.sql.Encoder[T])org.apache.spark.sql.Dataset[T].
  Unspecified value parameter evidence$7.
  lines.map(line =&gt; createRecord(line.split("\\|")))
  </code></pre>
  
  <p>I have a feeling this has something to do with implicits, TypeTags, and/or ClassTags, but I'm just starting out with Scala and don't fully understand these concepts yet.</p>