Answers.Body: <p>Slurm is killing the process because it's using more memory per core than the default allocation (that's what <code>slurmstepd: error: Exceeded step memory limit at some point</code> means). I've never used pyPiper (like seemingly most people, I use snakemake), but presuming that you have it running either <code>srun</code> or <code>sbatch</code> at some point then specify something like <code>--mem-per-cpu 10G</code>.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-12-15T15:12:39.587Z
Answers.Text: Actually I tried that and it still does not work. Will try to write to the cluster support
Answers.UserDisplayName: Nikita Vlasenko
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-12-15T15:22:35.497Z
Answers.Text: You might need a larger value than 10 gigs. You can use `scontrol` to see the exact threshold getting crossed that's causing slurm to kill the job.
Answers.UserDisplayName: Devon Ryan
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-12-15T16:53:31.197Z
Answers.Text: Awesome. We increased it to 20Gb and it run successfully.
Answers.UserDisplayName: Nikita Vlasenko
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-12-14T23:17:52.967Z
Answers.LastActivityDate: 2017-12-14T23:17:52.967Z
Answers.UserDisplayName: Devon Ryan
Body: <p>I am running a script file on a cluster with <code>slurm</code> through <code>sbatch</code>. Locally it runs perfectly well on 2 different linux versions (<code>linux mint</code>, <code>ubuntu 12.04</code>), however, it returns <code>-9</code> error code while running on a cluster which is <code>CentOS 7.3</code>. When we are using <code>Pypiper</code> and run the whole <code>.py</code> script file, it fails when it starts running <code>trimmomatic</code> program</p>
  
  <p><a href="http://www.usadellab.org/cms/?page=trimmomatic" rel="nofollow noreferrer">http://www.usadellab.org/cms/?page=trimmomatic</a></p>
  
  <p>However, when running just <code>trimmomatic</code> program step by itself it does not fail and finishes successfully. We have a suspicion that something may be wrong with how <code>Pypiper</code> runs on the cluster, however, feeling rather stuck.
  Any suggestions would be greatly appreciated. </p>
  
  <p>Here is the error that is printed out:</p>
  
  <pre><code>ILLUMINACLIP: Using 2 prefix pairs, 22 forward/reverse sequences, 0 
  forward only sequences, 0 reverse only sequences
  &lt;/pre&gt;
  Process 109391 returned: (-9). Elapsed: 0:00:51. Peak memory: (Process: 
  5.211GB; Pipeline: 0.0GB)
  
  ### Pipeline failed at:  (12-14 15:50:47) elapsed:0:23:09 _TIME_
  
  Total time: 0:23:09
  
  Change status from running to failed
  Traceback (most recent call last):
    File "...../salmon_rna_seq.py", line 344, in &lt;module&gt;
  sys.exit(main())
  File "...../salmon_rna_seq.py", line 304, in main
  pipeline.process()      
  File "...../salmon_rna_seq.py", line 277, in process
  self.trim_reads()      
  File "...../salmon_rna_seq.py", line 195, in trim_reads
  self.pipe_manager.run(cmd, self.sample_files['r1']['fastq']['trimmed_paired'])
  File "...../pypiper.py", line 508, in run
  process_return_code, local_maxmem = self.callprint(cmd, shell, nofail, container)  # Run command
  File "...../pypiper.py", line 691, in callprint
  self.fail_pipeline(e)
  File "...../pypiper.py", line 1111, in fail_pipeline
  raise e
  OSError: Subprocess returned nonzero result.
  
  Pypiper terminating spawned child process 108475...
  child process terminated
  slurmstepd: error: Exceeded step memory limit at some point.
  </code></pre>
  
  <p>The last line may be informative. Although, we are still not sure what we could possibly do.</p>
  
Tags: <rna-seq>
Title: Pipeline (Pypiper) returns error, but individual command works fine
ThreadText: # Pipeline (Pypiper) returns error, but individual command works fine
  
  (Asked by: username_0 on 2017-12-14)
  
  I am running a script file on a cluster with `slurm` through `sbatch`. Locally it runs perfectly well on 2 different linux versions (`linux mint`, `ubuntu 12.04`), however, it returns `-9` error code while running on a cluster which is `CentOS 7.3`. When we are using `Pypiper` and run the whole `.py` script file, it fails when it starts running `trimmomatic` program
  
  <http://www.usadellab.org/cms/?page=trimmomatic>
  
  However, when running just `trimmomatic` program step by itself it does not fail and finishes successfully. We have a suspicion that something may be wrong with how `Pypiper` runs on the cluster, however, feeling rather stuck.
  Any suggestions would be greatly appreciated.
  
  Here is the error that is printed out:
  
  ```
  ILLUMINACLIP: Using 2 prefix pairs, 22 forward/reverse sequences, 0 
  forward only sequences, 0 reverse only sequences
  </pre>
  Process 109391 returned: (-9). Elapsed: 0:00:51. Peak memory: (Process: 
  5.211GB; Pipeline: 0.0GB)
  
  ### Pipeline failed at:  (12-14 15:50:47) elapsed:0:23:09 _TIME_
  
  Total time: 0:23:09
  
  Change status from running to failed
  Traceback (most recent call last):
    File "...../salmon_rna_seq.py", line 344, in <module>
  sys.exit(main())
  File "...../salmon_rna_seq.py", line 304, in main
  pipeline.process()      
  File "...../salmon_rna_seq.py", line 277, in process
  self.trim_reads()      
  File "...../salmon_rna_seq.py", line 195, in trim_reads
  self.pipe_manager.run(cmd, self.sample_files['r1']['fastq']['trimmed_paired'])
  File "...../pypiper.py", line 508, in run
  process_return_code, local_maxmem = self.callprint(cmd, shell, nofail, container)  # Run command
  File "...../pypiper.py", line 691, in callprint
  self.fail_pipeline(e)
  File "...../pypiper.py", line 1111, in fail_pipeline
  raise e
  OSError: Subprocess returned nonzero result.
  
  Pypiper terminating spawned child process 108475...
  child process terminated
  slurmstepd: error: Exceeded step memory limit at some point.
  
  ```
  
  The last line may be informative. Although, we are still not sure what we could possibly do.
  
  **Tags:** rna-seq
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2017-12-14. Score: 3 | Accepted answer
  
  Slurm is killing the process because it's using more memory per core than the default allocation (that's what `slurmstepd: error: Exceeded step memory limit at some point` means). I've never used pyPiper (like seemingly most people, I use snakemake), but presuming that you have it running either `srun` or `sbatch` at some point then specify something like `--mem-per-cpu 10G`.
  
  > **Comments:**
  > **username_0**: Actually I tried that and it still does not work. Will try to write to the cluster support  
  > **username_1**: You might need a larger value than 10 gigs. You can use `scontrol` to see the exact threshold getting crossed that's causing slurm to kill the job.  
  > **username_0**: Awesome. We increased it to 20Gb and it run successfully.  
  
  ---
  
  