title: How to filter based on array value in PySpark?
body: <p>My Schema:</p>
  
  <pre class="lang-none prettyprint-override"><code>|-- Canonical_URL: string (nullable = true)
   |-- Certifications: array (nullable = true)
   |    |-- element: struct (containsNull = true)
   |    |    |-- Certification_Authority: string (nullable = true)
   |    |    |-- End: string (nullable = true)
   |    |    |-- License: string (nullable = true)
   |    |    |-- Start: string (nullable = true)
   |    |    |-- Title: string (nullable = true)
   |-- CompanyId: string (nullable = true)
   |-- Country: string (nullable = true)
  |-- vendorTags: array (nullable = true)
   |    |-- element: struct (containsNull = true)
   |    |    |-- score: double (nullable = true)
   |    |    |-- vendor: string (nullable = true)
  </code></pre>
  
  <p>I tried the below query to select nested fields from <code>vendorTags</code></p>
  
  <pre><code>df3 = sqlContext.sql("select vendorTags.vendor from globalcontacts")
  </code></pre>
  
  <p>How can I query the nested fields in <code>where</code> clause like below in PySpark</p>
  
  <pre><code>df3 = sqlContext.sql("select vendorTags.vendor from globalcontacts where vendorTags.vendor = 'alpha'")
  </code></pre>
  
  <p>or</p>
  
  <pre><code>df3 = sqlContext.sql("select vendorTags.vendor from globalcontacts where vendorTags.score &gt; 123.123456")
  </code></pre>
  
  <p>something like this..</p>
  
  <p>I tried the above queries only to get the below error</p>
  
  <pre><code>df3 = sqlContext.sql("select vendorTags.vendor from globalcontacts where vendorTags.vendor = 'alpha'")
  </code></pre>
  
  <pre class="lang-none prettyprint-override"><code>16/03/15 13:16:02 INFO ParseDriver: Parsing command: select vendorTags.vendor from globalcontacts where vendorTags.vendor = 'alpha'
  16/03/15 13:16:03 INFO ParseDriver: Parse Completed
  Traceback (most recent call last):
    File "&lt;stdin&gt;", line 1, in &lt;module&gt;
    File "/usr/lib/spark/python/pyspark/sql/context.py", line 583, in sql
      return DataFrame(self._ssql_ctx.sql(sqlQuery), self)
    File "/usr/lib/spark/python/lib/py4j-0.9-src.zip/py4j/java_gateway.py", line 813, in __call__
    File "/usr/lib/spark/python/pyspark/sql/utils.py", line 51, in deco
      raise AnalysisException(s.split(': ', 1)[1], stackTrace)
  pyspark.sql.utils.AnalysisException: u"cannot resolve '(vendorTags.vendor = cast(alpha as double))' due to data type mismatch: differing types in '(vendorTags.vendor = cast(alpha as double))' (array&lt;string&gt; and double).; line 1 pos 71"
  </code></pre>