Data Science, Analytics, Databricks, Data Cleaning.
removes duplicates where ALL the values in a row are the same. How about duplicates with the same user_id but different emails? For example, this user here: In fact, the user_id column has multiple rows with the same values: select count(distinct(user_id)) as distinct_user_ids, count(user_id) as