title: How to save dask dataframe to parquet on same machine as dask sheduler/workers?
body: <p>I'm trying to save by Dask Dataframe to parquet on the same machine as the dask scheduler/workers are located. However, I have trouble during this.</p>
  
  <p><strong>My Dask setup</strong>:
  My python script is executed on my local machine (laptop 16 GB RAM), but the script creates a Dask client to a Dask scheduler running on a remote machine (a server with 400 GB RAM for parallel computations). The Dask scheduler and workers are all located on the same server, thus they all share the same file system, locally available to them. As this remote Dask scheduler is used by all members of my team, the files we are working on are also located on the same server, providing common access to all members to all files through the same Dask cluster.</p>
  
  <p>I have tried:</p>
  
  <pre><code># This saves the parquet files in a folder on my local machine.
  ddf.to_parquet(
      '/scratch/dataset_no_dalayed', compression='brotli').compute()
  
  # This delayed call of `ddf.to_parquet` saves the Dask Dataframe chucks
  # into individual parquet files (i.e. parts) in the given folder.
  # However, I want to persist the Dask dataframe in my workflow, but this
  # fails as seen below.
  dask.delayed(ddf.to_parquet)(
      '/scratch/dataset_dalayed', compression='brotli').compute()
  
  # If the Dask dataframe is persisted, the `to_parquet` fails with
  # a "KilledWorker" error!
  ddf = client.persist(ddf)
  dask.delayed(ddf.to_parquet)(
      '/scratch/dataset_persist/', compression='brotli').compute()
  
  # In the example below, I can NOT save the Dask dataframe.
  # Because the delayed function makes the Dask dataframe
  # to a Pandas dataframe on runtime. And this fails as the path is a
  # folder and not at file as Pandas requires!
  @dask.delayed
  def save(new_ddf):
      new_ddf.to_parquet('/scratch/dataset_function/', compression='brotli')
  
  save(ddf).compute()
  </code></pre>
  
  <p>How to do this correct?</p>