Answers.Body: <p>What is happening is your Admin is not allowing you access to /scratch ... which is undertandable. Essentially you are attempting to access the headnode <code>/scratch</code>. If every user on the cluster had access, it would be a bit chaotic.</p>
  <p>I am surprised by the error, not that I would use Snakemake I'd code it, because you would expect a data pipeline tool as established as Snakemake to have a solution to working on a cluster. What you are describing is a standard cluster set-up between headnode and 'sub-node' (the technical term is slavenode, but I hate that term). The headnode is mirrored across the cluster, hence the error. Once you submit via <code>qsub</code> the mirror will 'display' the headnode layout on the subnode.</p>
  <ul>
  <li>You could ask your admin for permission to <code>/scratch</code>, this is root privilege. It is a bit unlikely they will say 'yes' but its worth a try.</li>
  <li>Otherwise what about a virtual environment? When Perl suffers this problem I simply use 'perlbrew', anaconda has very nice virtual environments that are easy to setup</li>
  <li>@Chris_Rand points out piping to a different output.</li>
  <li>I bet there is a solution in '.profile' and thats how I solved comparable problems in the past.</li>
  <li>Of course <code>Snakemake</code> do a Q&amp;A interaction on Overflow SE (but please come back here!) ... they must know the work around.</li>
  </ul>
  <p>Frankly I am astonished because the whole point of Snakemake, is the ease of constructing a data pipeline and data pipelines are processed on clusters, thus this error seems to up-end it, but who am I?</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-28T20:50:18.950Z
Answers.Text: Thanks, @Michael for the answer, but I don't think it solves really my problem. The `/scratch` directory does not exist on the head node at all and all the sub-nodes have thier local disks mapped there, to `/scrach`. I think the reason I am getting the errors is because snakemake is trying to verify that the directory exist at the frontend. That's quite strange to me, I would have guessed that the shadow directory will be created for every job individually when it's executed on the sub-node. I am not sure if I am clear.
Answers.UserDisplayName: Kamil S Jaron
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-28T20:55:06.610Z
Answers.Text: The issue is that the headnode is mirrored across the cluster, thus you have a copy of a file on the headnode - this file is not physically present on all 'subnodes' (slavenodes). Thus, because /scratch and permissions therein is not on the headnode this is present as soon as you run the queue 'qsub' or whatever.
Answers.UserDisplayName: M__
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-28T20:56:29.900Z
Answers.Text: If you physically logged into the subnode ('slavenode') there would be a /scratch (if you had permission to access it) and it would work. The thing is you access the subnode ('slavenode') via the mirrored queue.
Answers.UserDisplayName: M__
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-28T21:00:22.343Z
Answers.Text: Do you say that If I do `qsub <appropriate parameters> 'ls /scratch'` it will complain? Because I am pretty sure it would work.
Answers.UserDisplayName: Kamil S Jaron
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-28T21:01:35.340Z
Answers.Text: I am not sure if I follow the mirroring, but when I write non-snakemake jobs, I am using `/scratch` for temporary files all the time.
Answers.UserDisplayName: Kamil S Jaron
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-28T21:08:35.790Z
Answers.Text: Actually, even with snakemake jobs I am able to write to `/scratch` if I write it directly to the script. It's really the problem of using the functionality of `--shadow-prefix`. I might ask them more directly. P.S. sorry for non-being more concise
Answers.UserDisplayName: Kamil S Jaron
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-28T18:19:01.767Z
Answers.LastActivityDate: 2020-07-28T18:24:12.970Z
Answers.LastEditDate: 2020-07-28T18:24:12.970Z
Answers.UserDisplayName: M__
Body: <p>Snakemake is not really designed to store temporary files to local disk, but in the documentation, they suggest a <a href="https://snakemake.readthedocs.io/en/stable/project_info/faq.html#how-can-i-make-use-of-node-local-storage-when-running-cluster-jobs" rel="nofollow noreferrer">workaround using &quot;shadow rules&quot;</a>.</p>
  <p>The problem I have is, that our front node does not have <code>/scratch/..</code> directory and therefore snakemake complains right away</p>
  <pre><code>snakemake my_result.tsv -p --jobs 10 --shadow-prefix /scratch/$USER --cluster &quot;qsub &quot;
     ...
     PermissionError: [Errno 13] Permission denied: '/scratch'
  </code></pre>
  <p>Every single computational node does have the directory, only the frontend does not. Is there any easy workaround?</p>
  <p>--- edit ---</p>
  <p>I will try to clarify where I think is the problem.</p>
  <p>The head node does not have anything mounted to <code>/scratch</code>, I think that's why I am getting that error. I sort of expected that snakemake will attempt do create the shadow environments for individual executed jobs, because if it does, everything would be fine. All the computing nodes do have their local disks mounted to <code>/scratch</code>.</p>
  <p>Of course, I could use a different shadow directory, but all the directories that are mounted both at the front-end and computing nodes are NOT local disks of individual nodes, which would cause more network traveling of the temporary data.</p>
  <p>--- edit 2 ---</p>
  <p>Thanks @Chris_Rands for finding the relevant lines. I opened an <a href="https://github.com/snakemake/snakemake/issues/522" rel="nofollow noreferrer">Issue on Snakemake GitHub</a>. Once I will know more, I will get back with the answer here.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-28T14:59:42.597Z
Comments.Text: Not familiar with snakemake but, can you run the snakemake script from a node & reserve minimal resources for it as to not clog up your cluster? I suppose you might not be able to qsub while on a node.
Comments.UserDisplayName: Pallie
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-28T16:12:26.643Z
Comments.Text: Yeah, jobs can be submitted only from the frontend.
Comments.UserDisplayName: Kamil S Jaron
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-28T16:40:35.027Z
Comments.Text: can you set `--shadow-prefix` to a different directory path?
Comments.UserDisplayName: Chris_Rands
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-28T17:03:15.373Z
Comments.Text: It works when I specify a directory that is mounted on the frontend too
Comments.UserDisplayName: Kamil S Jaron
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-28T19:20:59.867Z
Comments.Text: What is the problem with just specifying another directory then? I don't feel this is a snakemake issue
Comments.UserDisplayName: Chris_Rands
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-28T20:37:08.867Z
Comments.Text: @Chris_Rands any other directory than `/scratch/...` is not a local directory on the computing nodes. If I pick a directory that is actually on the shared storage, all the temporary files will have to travel through networking. That's what I wanted to avoid with the shadow prefix.
Comments.UserDisplayName: Kamil S Jaron
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-29T09:59:38.883Z
Comments.Text: @KamilSJaron Ok, understood- they create the dir here https://github.com/snakemake/snakemake/blob/f945eef410fa31aa553631c5ec3b96c055f319c5/snakemake/persistence.py#L56-L72 but probably it's not needed until here https://github.com/snakemake/snakemake/blob/d550d20ce9520fd328e166938df253b2ca63fc8a/snakemake/jobs.py#L734-#L737
Comments.UserDisplayName: Chris_Rands
Tags: <snakemake>
Title: using snakemake shadow rules to store temp files on local nodes
ThreadText: # using snakemake shadow rules to store temp files on local nodes
  
  (Asked by: username_0 on 2020-07-28)
  
  Snakemake is not really designed to store temporary files to local disk, but in the documentation, they suggest a [workaround using "shadow rules"](https://snakemake.readthedocs.io/en/stable/project_info/faq.html#how-can-i-make-use-of-node-local-storage-when-running-cluster-jobs).
  
  The problem I have is, that our front node does not have `/scratch/..` directory and therefore snakemake complains right away
  
  ```
  snakemake my_result.tsv -p --jobs 10 --shadow-prefix /scratch/$USER --cluster "qsub "
     ...
     PermissionError: [Errno 13] Permission denied: '/scratch'
  
  ```
  
  Every single computational node does have the directory, only the frontend does not. Is there any easy workaround?
  
  --- edit ---
  
  I will try to clarify where I think is the problem.
  
  The head node does not have anything mounted to `/scratch`, I think that's why I am getting that error. I sort of expected that snakemake will attempt do create the shadow environments for individual executed jobs, because if it does, everything would be fine. All the computing nodes do have their local disks mounted to `/scratch`.
  
  Of course, I could use a different shadow directory, but all the directories that are mounted both at the front-end and computing nodes are NOT local disks of individual nodes, which would cause more network traveling of the temporary data.
  
  --- edit 2 ---
  
  Thanks @Chris\_Rands for finding the relevant lines. I opened an [Issue on Snakemake GitHub](https://github.com/snakemake/snakemake/issues/522). Once I will know more, I will get back with the answer here.
  
  **Tags:** snakemake
  
  > **Comments:**
  > **username_1**: Not familiar with snakemake but, can you run the snakemake script from a node & reserve minimal resources for it as to not clog up your cluster? I suppose you might not be able to qsub while on a node.  
  > **username_0**: Yeah, jobs can be submitted only from the frontend.  
  > **username_2**: can you set `--shadow-prefix` to a different directory path?  
  > **username_0**: It works when I specify a directory that is mounted on the frontend too  
  > **username_2**: What is the problem with just specifying another directory then? I don't feel this is a snakemake issue  
  > **username_0**: @username_2 any other directory than `/scratch/...` is not a local directory on the computing nodes. If I pick a directory that is actually on the shared storage, all the temporary files will have to travel through networking. That's what I wanted to avoid with the shadow prefix.  
  > **username_2**: @username_0 Ok, understood- they create the dir here https://github.com/snakemake/snakemake/blob/f945eef410fa31aa553631c5ec3b96c055f319c5/snakemake/persistence.py#L56-L72 but probably it's not needed until here https://github.com/snakemake/snakemake/blob/d550d20ce9520fd328e166938df253b2ca63fc8a/snakemake/jobs.py#L734-#L737  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2020-07-28. Score: 0
  
  What is happening is your Admin is not allowing you access to /scratch ... which is undertandable. Essentially you are attempting to access the headnode `/scratch`. If every user on the cluster had access, it would be a bit chaotic.
  
  I am surprised by the error, not that I would use Snakemake I'd code it, because you would expect a data pipeline tool as established as Snakemake to have a solution to working on a cluster. What you are describing is a standard cluster set-up between headnode and 'sub-node' (the technical term is slavenode, but I hate that term). The headnode is mirrored across the cluster, hence the error. Once you submit via `qsub` the mirror will 'display' the headnode layout on the subnode.
  
  * You could ask your admin for permission to `/scratch`, this is root privilege. It is a bit unlikely they will say 'yes' but its worth a try.
  * Otherwise what about a virtual environment? When Perl suffers this problem I simply use 'perlbrew', anaconda has very nice virtual environments that are easy to setup
  * @Chris\_Rand points out piping to a different output.
  * I bet there is a solution in '.profile' and thats how I solved comparable problems in the past.
  * Of course `Snakemake` do a Q&A interaction on Overflow SE (but please come back here!) ... they must know the work around.
  
  Frankly I am astonished because the whole point of Snakemake, is the ease of constructing a data pipeline and data pipelines are processed on clusters, thus this error seems to up-end it, but who am I?
  
  > **Comments:**
  > **username_0**: Thanks, @Michael for the answer, but I don't think it solves really my problem. The `/scratch` directory does not exist on the head node at all and all the sub-nodes have thier local disks mapped there, to `/scrach`. I think the reason I am getting the errors is because snakemake is trying to verify that the directory exist at the frontend. That's quite strange to me, I would have guessed that the shadow directory will be created for every job individually when it's executed on the sub-node. I am not sure if I am clear.  
  > **username_3**: The issue is that the headnode is mirrored across the cluster, thus you have a copy of a file on the headnode - this file is not physically present on all 'subnodes' (slavenodes). Thus, because /scratch and permissions therein is not on the headnode this is present as soon as you run the queue 'qsub' or whatever.  
  > **username_3**: If you physically logged into the subnode ('slavenode') there would be a /scratch (if you had permission to access it) and it would work. The thing is you access the subnode ('slavenode') via the mirrored queue.  
  > **username_0**: Do you say that If I do `qsub <appropriate parameters> 'ls /scratch'` it will complain? Because I am pretty sure it would work.  
  > **username_0**: I am not sure if I follow the mirroring, but when I write non-snakemake jobs, I am using `/scratch` for temporary files all the time.  
  > **username_0**: Actually, even with snakemake jobs I am able to write to `/scratch` if I write it directly to the script. It's really the problem of using the functionality of `--shadow-prefix`. I might ask them more directly. P.S. sorry for non-being more concise  
  
  ---
  
  