Does somebody know why these improvements could not be integrated into DataFrames directly?<p>Why add another very similar layer of abstraction?<p>They mention:<p>&quot;Unification of DataFrames with Datasets – due to compatibility guarantees, DataFrames and Datasets currently cannot share a common parent class. With Spark 2.0, we will be able to unify these abstractions with minor changes to the API, making it easy to build libraries that work with both.&quot;<p>Anybody know what those &quot;compatibility guarantees&quot; are?