title: Conditional Spark map() function based on input columns
body: <p>What I'm trying to achieve here is sending to Spark SQL <code>map</code> function conditionally generated columns depending on if they have <code>null</code>, <code>0</code> or any other value I may want.</p>
  
  <p>Take for example this initial DF.</p>
  
  <pre><code>val initialDF = Seq(
    ("a", "b", 1), 
    ("a", "b", null), 
    ("a", null, 0)
  ).toDF("field1", "field2", "field3")
  </code></pre>
  
  <p>From that initial DataFrame I want to generate yet another column which will be a map, like this.</p>
  
  <pre class="lang-scala prettyprint-override"><code>initialDF.withColumn("thisMap", MY_FUNCTION)
  </code></pre>
  
  <p>My current approach to this is basically take a <code>Seq[String]</code> in a method a <code>flatMap</code> the key-value pairs that the Spark SQL method receives, like this.</p>
  
  <pre class="lang-scala prettyprint-override"><code>def toMap(columns: String*): Column = {
    map(
      columns.flatMap(column =&gt; List(lit(column), col(column))): _*
    )
  }
  </code></pre>
  
  <p>But then, filtering becomes a Scala thing and is quite a mess.</p>
  
  <p>What I would like to obtain after the processing would be, for each of those rows, the next DataFrame.</p>
  
  <pre><code>val initialDF = Seq(
    ("a", "b", 1, Map("field1" -&gt; "a", "field2" -&gt; "b", "field3" -&gt; 1)),
    ("a", "b", null, Map("field1" -&gt; "a", "field2" -&gt; "b")),
    ("a", null, 0, Map("field1" -&gt; "a"))
  )
    .toDF("field1", "field2", "field3", "thisMap")
  </code></pre>
  
  <p>I was wondering if this can be achieved using the <code>Column</code> API which is way more intuitive with <code>.isNull</code> or <code>.equalTo</code>?</p>