title: Passing a data frame column and external list to udf under withColumn
body: <p>I have a Spark dataframe with the following structure. The bodyText_token has the tokens (processed/set of words). And I have a nested list of defined keywords</p>
  <pre><code>root
   |-- id: string (nullable = true)
   |-- body: string (nullable = true)
   |-- bodyText_token: array (nullable = true)
  
  keyword_list=[['union','workers','strike','pay','rally','free','immigration',],
  ['farmer','plants','fruits','workers'],['outside','field','party','clothes','fashions']]
  </code></pre>
  <p>I needed to check how many tokens fall under each keyword list and add the result as a new column of the existing dataframe.
  Eg: if <code>tokens =[&quot;become&quot;, &quot;farmer&quot;,&quot;rally&quot;,&quot;workers&quot;,&quot;student&quot;]</code>
  the result will be -&gt; <code>[1,2,0]</code></p>
  <p>The following function worked as expected.</p>
  <pre><code>def label_maker_topic(tokens,topic_words):
      twt_list = []
      for i in range(0, len(topic_words)):
          count = 0
          #print(topic_words[i])
          for tkn in tokens:
              if tkn in topic_words[i]:
                  count += 1
          twt_list.append(count)
      
      return twt_list
  </code></pre>
  <p>I used udf under <code>withColumn</code> to access the function and I get an error. I think it's about passing an external list to a udf. Is there a way I can pass the external list and the dataframe column to a udf and add a new column to my dataframe?</p>
  <pre><code>topicWord = udf(label_maker_topic,StringType())
  myDF=myDF.withColumn(&quot;topic_word_count&quot;,topicWord(myDF.bodyText_token,keyword_list))
  </code></pre>