Question: I've read that Glacier can store documents for "month, years and even decades".
Where or how I can specify how long archive needed to be stored at Amazon?

Answer: 
The time stamp column doesn't exist yet when you try to refer to it; You can either use `pyspark.sql.functions.col` to refer to it in a dynamic way without specifying which data frame object the column belongs to as:

```
import pyspark.sql.functions as F

df = df.withColumn("unix_timestamp", df.unix_timestamp.cast("timestamp")).filter(F.col("unix_timestamp") > hours_36)

```

Or without creating the intermediate column:

```
df.filter(df.unix_timestamp.cast("timestamp") > hours_36)

```
The API Doc tells me that you can also use a String notation for filtering:
<https://spark.apache.org/docs/latest/api/python/pyspark.sql.html#pyspark.sql.DataFrame.filter>

```
    import pyspark.sql.functions as F

df = df.withColumn("unix_timestamp", df.unix_timestamp.cast("timestamp"))
    .filter("unix_timestamp > %s" % hours_36)

```

Maybe its not so effienc though