title: Create local_directory for dask_jobqueue
body: <p>I'm trying to run dask on an HPC system that uses NFS for storage. As such, I want to configure dask to use local storage for scratch space. Each cluster node has a <code>/scratch/</code> folder that all users can write into, with instructions to put scratch files in <code>/scratch/&lt;username&gt;/&lt;jobid&gt;/</code>.</p>
  <p>I have a bit of code configured as such:</p>
  <pre class="lang-py prettyprint-override"><code>import dask_jobqueue
  from distributed import Client
  
  cluster = dask_jobqueue.SLURMCluster(
              queue = 'high',
              cores = 24,
              memory = '60GB',
              walltime = '10:00:00',
              local_directory = '/scratch/&lt;username&gt;/&lt;jobid&gt;/'
  )
  
  cluster.scale(1)
  client = Client(cluster)
  </code></pre>
  <p>However, I have a problem. The directory doesn't exist (both because I don't know what node the client will be on, and because it creates it based on the SLURM job id, which is always unique) and so my code fails with:</p>
  <pre><code>Process Dask Worker process (from Nanny):
  Traceback (most recent call last):
    File &quot;/home/lsterzin/anaconda3/lib/python3.7/multiprocessing/process.py&quot;, line 297, in _bootstrap
      self.run()
    File &quot;/home/lsterzin/anaconda3/lib/python3.7/multiprocessing/process.py&quot;, line 99, in run
      self._target(*self._args, **self._kwargs)
    File &quot;/home/lsterzin/anaconda3/lib/python3.7/site-packages/distributed/process.py&quot;, line 191, in _run
      target(*args, **kwargs)
    File &quot;/home/lsterzin/anaconda3/lib/python3.7/site-packages/distributed/nanny.py&quot;, line 699, in _run
      worker = Worker(**worker_kwargs)
    File &quot;/home/lsterzin/anaconda3/lib/python3.7/site-packages/distributed/worker.py&quot;, line 497, in __init__
      self._workspace = WorkSpace(os.path.abspath(local_directory))
    File &quot;/home/lsterzin/anaconda3/lib/python3.7/site-packages/distributed/diskutils.py&quot;, line 118, in __init__
      self._init_workspace()
    File &quot;/home/lsterzin/anaconda3/lib/python3.7/site-packages/distributed/diskutils.py&quot;, line 124, in _init_workspace
      os.mkdir(self.base_dir)
  FileNotFoundError: [Errno 2] No such file or directory: '/scratch/&lt;user&gt;/&lt;jobid&gt;'
  </code></pre>
  <p>I can't create the directory without knowing what node the dask workers will be running on, and I can't create the cluster with <code>dask_jobqueue</code> without the directory already existing. What's the best way to work around this?</p>