title: Load Spark data locally Incomplete HDFS URI
body: <p>I have experienced a problem with SBT loading in a local CSV file. Basically, I've written a Spark program in Scala Eclipse which reads the following file:</p>
  
  <pre><code>val searches = sc.textFile("hdfs:///data/searches")
  </code></pre>
  
  <p>This works fine on hdfs, but for de-bug reasons, I wish to load in this file from a local directory, which I have set-up to be in the project directory. </p>
  
  <p>So I tired the following:</p>
  
  <pre><code>val searches = sc.textFile("file:///data/searches")
  val searches = sc.textFile("./data/searches")
  val searches = sc.textFile("/data/searches")
  </code></pre>
  
  <p>None of which allows me to read the file from local, and all of them returns this error on SBT:</p>
  
  <pre><code>Exception in thread "main" java.io.IOException: Incomplete HDFS URI, no host: hdfs:/data/pages
  at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:143)
  at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2397)
  at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:89)
  at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2431)
  at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2413)
  at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:368)
  at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
  at org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:256)
  at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:228)
  at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:304)
  at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:179)
  at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:204)
  at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:202)
  at scala.Option.getOrElse(Option.scala:120)
  at org.apache.spark.rdd.RDD.partitions(RDD.scala:202)
  at org.apache.spark.rdd.MappedRDD.getPartitions(MappedRDD.scala:28)
  at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:204)
  at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:202)
  at scala.Option.getOrElse(Option.scala:120)
  at org.apache.spark.rdd.RDD.partitions(RDD.scala:202)
  at org.apache.spark.rdd.FlatMappedRDD.getPartitions(FlatMappedRDD.scala:30)
  at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:204)
  at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:202)
  at scala.Option.getOrElse(Option.scala:120)
  at org.apache.spark.rdd.RDD.partitions(RDD.scala:202)
  at org.apache.spark.rdd.MappedRDD.getPartitions(MappedRDD.scala:28)
  at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:204)
  at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:202)
  at scala.Option.getOrElse(Option.scala:120)
  at org.apache.spark.rdd.RDD.partitions(RDD.scala:202)
  at org.apache.spark.SparkContext.runJob(SparkContext.scala:1135)
  at org.apache.spark.rdd.RDD.count(RDD.scala:904)
  at com.user.Result$.get(SparkData.scala:200)
  at com.user.StreamingApp$.main(SprayHerokuExample.scala:35)
  at com.user.StreamingApp.main(SprayHerokuExample.scala)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:606)
  at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:328)
  at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
  at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
  </code></pre>
  
  <p>In the error report, at com.user.Result$.get(SparkData.scala:200) is the line where sc.textFile is called. It seems to run in Hadoop environment by default. Is there anything I could do to read this file locally?</p>
  
  <p>Edit: While on Local, I've reconfigured build.sbt with:</p>
  
  <pre><code>submit &lt;&lt;= inputTask{(argTask:TaskKey[Seq[String]]) =&gt; {
  (argTask,mainClass in Compile,assemblyOutputPath in assembly,sparkHome) map { 
  (args,main,jar,sparkHome) =&gt; {
    args match {
      case List(output) =&gt; {
        val sparkCmd = sparkHome+"/bin/spark-submit"
        Process(
          sparkCmd :: "--class" :: main.get :: "--master" :: "local[4]" ::
          jar.getPath :: "local[4]" :: output :: Nil)!
      } 
      case _ =&gt; Process("echo" :: "Usage" :: Nil) !
    }
  }
  </code></pre>
  
  <p>}}}</p>
  
  <p>The submit command is what I use to run the code. </p>
  
  <p>Solution Found: So it turns out that file:///path/ is the correct way to do it, but in my case, the full path worked: i.e. home/projects/data/searches. While just putting data/searches did not (despite working under home/projects directory).</p>