Question: I want to filter a Pyspark DataFrame with a SQL-like `IN` clause, as in

```
sc = SparkContext()
sqlc = SQLContext(sc)
df = sqlc.sql('SELECT * from my_df WHERE field1 IN a')

```

where `a` is the tuple `(1, 2, 3)`. I am getting this error:

> 
> java.lang.RuntimeException: [1.67] failure: ``('' expected but identifier a found
> 
> 
> 

which is basically saying it was expecting something like *'(1, 2, 3)'* instead of a.
The problem is I can't manually write the values in a as it's extracted from another job.

How would I filter in this case?

Answer: 
Just a little addition/update:

```
choice_list = ["foo", "bar", "jack", "joan"]

```

If you want to filter your dataframe "df", such that you want to keep rows based upon a column "v" taking only the values from choice\_list, then

```
from pyspark.sql.functions import col

df_filtered = df.where( ( col("v").isin (choice_list) ) )

```
```python
from pyspark.sql import SparkSession
import pandas as pd
spark=SparkSession.builder.appName('Practise').getOrCreate()
df_pyspark=spark.read.csv('datasets/myData.csv',header=True,inferSchema=True)
df_spark.createOrReplaceTempView("df") # we need to create a Temp table first
spark.sql("SELECT * FROM df where Departments in ('IOT','Big Data') order by Departments").show()

```