For me, the biggest improvement is the unified typed Dataset API [1]. The current Dataset API gave us a lot of flexibility and type-safety and the new API lets us use it as DataFrame API instead of converting to RDD and reinventing the wheel, like aggregators [2].<p>[1] <a href="https:&#x2F;&#x2F;databricks-prod-cloudfront.cloud.databricks.com&#x2F;public&#x2F;4027ec902e239c93eaaa8714f173bcfc&#x2F;6122906529858466&#x2F;431554386690871&#x2F;4814681571895601&#x2F;latest.html" rel="nofollow">https:&#x2F;&#x2F;databricks-prod-cloudfront.cloud.databricks.com&#x2F;publ...</a><p>[2] <a href="https:&#x2F;&#x2F;docs.cloud.databricks.com&#x2F;docs&#x2F;spark&#x2F;1.6&#x2F;examples&#x2F;Dataset%20Aggregator.html" rel="nofollow">https:&#x2F;&#x2F;docs.cloud.databricks.com&#x2F;docs&#x2F;spark&#x2F;1.6&#x2F;examples&#x2F;Da...</a>