Stack Exchange
How to use decision tree with dataset from CSV file?
I'd like to use Spark MLlib's org.apache.spark.mllib.tree.DecisionTree as in the following code, but compilation fails.
import org.apache.spark.ml.Pipeline
import org.apache.spark.ml.classification.DecisionTreeClassifier
import org.apache.spark.ml.classification.DecisionTreeClassificationModel
import org.apache.spark.ml.feature.{StringIndexer, IndexToString, VectorIndexer}
import org.apache.spark.ml.evaluation.MulticlassClassificationEvaluator
import org.apache.spark.mllib.tree.DecisionTree
import org.apache.spark.mllib.tree.model.DecisionTreeModel
import org.apache.spark.mllib.util.MLUtils
import org.apache.spark.sql.SparkSession

val sqlContext = new org.apache.spark.sql.SQLContext(sc)
val data = sqlContext.read.format("csv").load("C:/spark/spark-2.1.0-bin-hadoop2.7/data/mllib/airlines.txt")
val df = sqlContext.read.csv("C:/spark/spark-2.1.0-bin-hadoop2.7/data/mllib/airlines.txt")
val dataframe = sqlContext.createDataFrame(df).toDF("label");
val splits = data.randomSplit(Array(0.7, 0.3))

val (trainingData, testData) = (splits(0), splits(1))

val numClasses = 2
val categoricalFeaturesInfo = Map[Int, Int]()
val impurity = "gini"
val maxDepth = 5
val maxBins = 32
val model = DecisionTree.trainClassifier(trainingData, numClasses, categoricalFeaturesInfo,impurity, maxDepth, maxBins)

Compilation fails with the following error message:

<console>:44: error: overloaded method value trainClassifier with
  alternatives:   (input:
  org.apache.spark.api.java.JavaRDD[org.apache.spark.mllib.regression.LabeledPoint],numClasses:
  Int,categoricalFeaturesInfo: java.util.Map[Integer,Integer],impurity:
  String,maxDepth: Int,maxBins:
  Int)org.apache.spark.mllib.tree.model.DecisionTreeModel 
  (input:
  org.apache.spark.rdd.RDD[org.apache.spark.mllib.regression.LabeledPoint],numClasses:
  Int,categoricalFeaturesInfo:
  scala.collection.immutable.Map[Int,Int],impurity: String,maxDepth:
  Int,maxBins: Int)org.apache.spark.mllib.tree.model.DecisionTreeModel 
  cannot be applied to
  (org.apache.spark.sql.Dataset[org.apache.spark.sql.Row], Int,
  scala.collection.immutable.Map[Int,Int], String, Int, Int)
         val model = DecisionTree.trainClassifier(trainingData, numClasses, categoricalFeaturesInfo,impurity, maxDepth, maxBins)


I am getting this error "overloaded method value trainClassifier with alternatives " when I am running the above code. It would be great if you can solve this issue.

You use the old RDD-based DecisionTree with Spark SQL's new Dataset API and hence the compilation error:

cannot be applied to (org.apache.spark.sql.Dataset[org.apache.spark.sql.Row], Int, scala.collection.immutable.Map[Int,Int], String, Int, Int) val model = DecisionTree.trainClassifier(trainingData, numClasses, categoricalFeaturesInfo,impurity, maxDepth, maxBins)

Note the first input parameter that is of type org.apache.spark.sql.Dataset[org.apache.spark.sql.Row], but DecisionTree requires org.apache.spark.api.java.JavaRDD[org.apache.spark.mllib.regression.LabeledPoint].
Quoting Announcement: DataFrame-based API is primary API:

As of Spark 2.0, the RDD-based APIs in the spark.mllib package have entered maintenance mode. The primary Machine Learning API for Spark is now the DataFrame-based API in the spark.ml package.

Please change your code according to Decision trees:

The spark.ml implementation supports decision trees for binary and multiclass classification and for regression, using both continuous and categorical features. The implementation partitions data by rows, allowing distributed training with millions or even billions of instances.

