Question: I have an array as a broadcast variable and it contains Integers: 

```
broadcast_array.value
Array(72159153, 72159163, 72159202, 72159203, 72159238, 72159398, 72159447, 72159448, 72159455, 72159492...

```

I have a column in a dataset (call is `col_id` which contains `IntegerType` values that might be in the `broadcast_array`, but they might not.

I am only trying to create a new column (call it `new_col`) that checks if `col_id` value for each row is in `broadcast_array`. If so, the new column value should be `Available`, else it can be `null`

So I have something like:

```
val my_new_df = df.withColumn("new_col", when(broadcast_array.value.contains($"col_id"), "Available"))

```

But I keep getting this error: 

```
Name: Unknown Error
Message: <console>:45: error: type mismatch;
 found   : Boolean
 required: org.apache.spark.sql.Column
   val my_new_df = df.withColumn("new_col", when(broadcast_array.value.contains($"col_id"), "Available"))
                                                                                           ^
StackTrace: 

```

What is most confusing to me is that I thought the `when` statement requires a conditional that outputs some Boolean, but here it's saying it requires a Column. 

How should I go about adding a value to a new column based on whether the value in an existing column can be found in a predefined Array or not?

Answer: 
If you look at the *api* of `when` function

> 
> def when(condition : org.apache.spark.sql.Column, value : scala.Any) : org.apache.spark.sql.Column
> 
> 
> 

Its clear that **the condition required is a column and not a boolean**.

So you can do complicated `lit` combinations to convert your `boolean` to `column` as 

```
import org.apache.spark.sql.functions._
df.withColumn("new_col", when(lit(broadcast_array.value.mkString(",")).contains($"col_id"), lit("Available"))).show(false)

```

OR 

You can achieve what you are trying by writing a simple `udf` function as 

```
import org.apache.spark.sql.functions._
val broadcastContains = udf((id: Int) => broadcast_array.value.contains(id))

```

and just call the function as 

```
df.withColumn("new_col", when(broadcastContains($"col_id"), lit("Available"))).show(false)

```
I added a `broadcastArrayContains` function to [spark-daria](https://github.com/MrPowers/spark-daria) that makes Ramesh's solution more reusable / accessible.

```
def broadcastArrayContains[T](col: Column, broadcastedArray: Broadcast[Array[T]]) = {
  when(col.isNull, null)
    .when(lit(broadcastedArray.value.mkString(",")).contains(col), lit(true))
    .otherwise(lit(false))
}

```

Suppose you have the following DataFrame (`df`):

```
+----+
| num|
+----+
| 123|
|  hi|
|null| 
+----+

```

You can identify all the values in the broadcasted array as follows:

```
val specialNumbers = spark.sparkContext.broadcast(Array("123", "456"))

df.withColumn(
  "is_special_number",
  functions.broadcastArrayContains[String](col("num"), specialNumbers)
)

```