title: Identifying why data is skewed in Spark
body: <p>I am investigating a Spark SQL job (Spark 1.6.0) that is performing poorly due to badly skewed data across the 200 partitions, most of the data is in 1 partition:
  <a href="https://i.stack.imgur.com/PDHZt.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/PDHZt.png" alt="Index,ID,Attempt,Status,Locality Level,Executor ID/Host,Launch Time,Duration,GC Time,Shuffle Read Size/Records"></a>
  What I'm wondering is...is there anything in the Spark UI to help me find out more about how the data is partitioned? From looking at this I don't know which columns the dataframe is partitioned on. How can I find that out? (other than looking at the code - I'm wondering if there's anything in the logs and/or UI that could help me)?</p>
  
  <p>Additional details, this is using Spark's dataframe API, Spark version 1.6. Underlying data is stored in parquet format.</p>