title: Strategy for partitioning dask dataframes efficiently
body: <p>The documentation for Dask talks about repartioning to reduce overhead <a href="https://docs.dask.org/en/latest/dataframe-best-practices.html#repartition-to-reduce-overhead" rel="noreferrer">here</a>.</p>
  
  <p>They however seem to indicate you need some knowledge of what your dataframe will look like beforehand (ie that there will 1/100th of the data expected).</p>
  
  <p>Is there a good way to repartition sensibly without making assumptions? At the moment I just repartition with <code>npartitions = ncores * magic_number</code>, and set force to <code>True</code> to expand partitions if need be. This one size fits all approach works but is definitely suboptimal as my dataset varies in size.</p>
  
  <p>The data is time series data, but unfortunately not at regular intervals, I've used repartition by time frequency in the past but this would be suboptimal because of how irregular the data is (sometimes nothing for minutes then thousands in seconds)</p>