Data Governance, Data Warehouse, Databricks, Databricks Sql, Data Observability.
for future analysis queries_df_with_window = queries_df.withColumn("start_window", lit(current_time_sub_hour_ms)).withColumn("end_window", lit(current_time_ms)) #append the dataframe to a table to be used for alerts and analysis