title: Data transfer from csv format to Redis hash format in Databricks
body: <p>I have an Azure system divided into three parts:</p>
  <ol>
  <li>Azure Data lake storage where I have some csv file.</li>
  <li>Azure Databricks where I need to make some process - exactly is to convert that csv file to Redis hash format.</li>
  <li>Azure Redis cache where I should put that converted data.</li>
  </ol>
  <p>After mounting storage in databricks filesystem there is a need to process some data.
  How to convert csv data located in databricks filesystem to redisHash format and correctly to put it to Redis?
  Specifically, I'm not sure how to make a correct mapping by the following code below. Or maybe is there some way of additional transfer to SQL table which I cannot find.</p>
  <p>Here is my example of code written on scala:</p>
  <pre><code>import com.redislabs.provider.redis._
  
  val redisServerDnsAddress = &quot;HOST&quot;
  val redisPortNumber = 6379
  val redisPassword = &quot;Password&quot;
  val redisConfig = new RedisConfig(new RedisEndpoint(redisServerDnsAddress, redisPortNumber, redisPassword))
  
  
  val data = spark.read.format(&quot;com.databricks.spark.csv&quot;).option(&quot;header&quot;, &quot;true&quot;).option(&quot;inferSchema&quot;, &quot;true&quot;).load(&quot;/mnt/staging/data/file.csv&quot;)
  
  // What is the right way of mapping?
  val ds = table(&quot;data&quot;).select(&quot;Prop1&quot;, &quot;Prop2&quot;, &quot;Prop3&quot;, &quot;Prop4&quot;, &quot;Prop5&quot; ).distinct.na.drop().map{x =&gt;
    (x.getString(0), x.getString(1), x.getString(2), x.getString(3), x.getString(4))
  }
  
  sc.toRedisHASH(ds, &quot;data&quot;)
  </code></pre>
  <p>The error:</p>
  <pre><code>error: type mismatch;
   found   : org.apache.spark.sql.Dataset[(String, String)]
   required: org.apache.spark.rdd.RDD[(String, String)]
  sc.toRedisHASH(ds, &quot;data&quot;)
  </code></pre>
  <p>If I write the last string of code this way:</p>
  <pre><code>sc.toRedisHASH(ds.rdd, &quot;data&quot;)
  </code></pre>
  <p>The error:</p>
  <pre><code>org.apache.spark.sql.AnalysisException: Table or view not found: data;
  </code></pre>