Answers.Body: <p>Looper does not generate any of these per-sample outputs.</p>
  
  <p>Looper does not have any control over where your pipeline puts its outputs.</p>
  
  <p>The problem is in your pipeline, not in looper. Assuming you're using a pypiper pipeline, I suggest reading the docs; for example, the "your first pipeline" tutorial: <a href="http://pypiper.readthedocs.io/en/latest/tutorials-first.html" rel="nofollow noreferrer">http://pypiper.readthedocs.io/en/latest/tutorials-first.html</a></p>
  
  <p>For real-life example, see <a href="https://github.com/databio/dnameth_pipelines/blob/3e6abd03185f8e2d5ef1c72849afdf9e5033f269/src/rrbs.py" rel="nofollow noreferrer">this pipeline</a></p>
  
  <p><code>
  outfolder = os.path.abspath(os.path.join(args.output_parent, args.sample_name))
  pm = pypiper.PipelineManager(name="RRBS", outfolder=outfolder, args=args, version=__version__)
  </code></p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-01-20T03:29:58.567Z
Answers.LastActivityDate: 2018-01-20T03:29:58.567Z
Answers.UserDisplayName: nsheff
Body: <p>I have multiple samples that I need to run one pipeline on. I am using Looper: <a href="https://github.com/pepkit/looper" rel="nofollow noreferrer">https://github.com/pepkit/looper</a></p>
  
  <p>I have a real trouble specifying a separate location for each pipeline output files. The first time when I did not care about it thinking that <code>Looper</code> would just automatically do that, it lead to multiple files being overwritten. Then, trying to avoid it I tried (and it did not work):</p>
  
  <blockquote>
    <ol>
    <li>Adding <code>{sample_name}</code> to the <code>output_dir</code> of the <code>metadata</code> section of <code>project_config.yaml</code> file:</li>
    </ol>
  </blockquote>
  
  <pre><code> metadata:
    sample_annotation: sandro_annotation.csv
    output_dir: /scratch/nv4e/sandro_looper_results/{sample_name}
    pipelines_dir: [../pipeline/pipeline_interface.yaml]
  </code></pre>
  
  <blockquote>
    <ol start="2">
    <li>Creating a <code>derived_column</code> in the <code>.csv</code> annotation file:</li>
    </ol>
  </blockquote>
  
  <pre><code>sample_name,library,sample_name_1_read,sample_name_2_read,output_folder
  abeta_24h_1,RNA-seq,read1,read2,out
  abeta_24h_2,RNA-seq,read1,read2,out
  abeta_24h_3,RNA-seq,read1,read2,out
  </code></pre>
  
  <p><code>pipeline_interface.yaml</code>:</p>
  
  <pre><code>...
  pipelines:
    sandro_rna_seq:
      name: sandro_rna_seq
      path: sandro_rna_seq.py  # relative to this 
  pipeline_interface.yaml file
      looper_args: True
      arguments:
        "-S": sample_name
        "--input": sample_name_1_read
        "-O": output_folder
  ...
  </code></pre>
  
  <p>and <code>project.config.yaml</code>:</p>
  
  <pre><code>...
  derived_columns: [sample_name_1_read,sample_name_2_read,output_folder]
  data_sources:
    read1: /scratch/nv4e/sandro_looper/data/merged/{sample_name}_r1.fq.gz
    read2: /scratch/nv4e/sandro_looper/data/merged/{sample_name}_r2.fq.gz
    out: /scratch/nv4e/sandro_looper_results/results_pipeline/{sample_name}
  ...
  </code></pre>
  
  <p>In this particular case the parameter <code>-O</code> was formed correctly which I was able to see in <code>sandro_looper_results/submissions/</code> <code>.sub</code> file, however, it was supplied the second time, apparently from the <code>output_dir</code> section of the <code>project_config.yaml</code> and was overriding the first <code>-O</code> command. So, obviously I tried to delete <code>output_dir</code> line but it is a required setting for <code>Looper</code>, so it can not run without it.</p>
  
  <blockquote>
    <ol start="3">
    <li>I have also tried supplying <code>-O</code> command in <code>project_config.yaml</code> in the <code>pipeline_args:</code> section with no success.</li>
    </ol>
  </blockquote>
  
  <p>I really do not know what I could do at this point. Any suggestions would be greatly appreciated,</p>
  
Tags: <scrnaseq><yaml><looper>
Title: Looper: can't specify the individual path for one pipeline run
ThreadText: # Looper: can't specify the individual path for one pipeline run
  
  (Asked by: username_0 on 2018-01-20)
  
  I have multiple samples that I need to run one pipeline on. I am using Looper: <https://github.com/pepkit/looper>
  
  I have a real trouble specifying a separate location for each pipeline output files. The first time when I did not care about it thinking that `Looper` would just automatically do that, it lead to multiple files being overwritten. Then, trying to avoid it I tried (and it did not work):
  
  > 1. Adding `{sample_name}` to the `output_dir` of the `metadata` section of `project_config.yaml` file:
  
  ```
   metadata:
    sample_annotation: sandro_annotation.csv
    output_dir: /scratch/nv4e/sandro_looper_results/{sample_name}
    pipelines_dir: [../pipeline/pipeline_interface.yaml]
  
  ```
  
  > 2. Creating a `derived_column` in the `.csv` annotation file:
  
  ```
  sample_name,library,sample_name_1_read,sample_name_2_read,output_folder
  abeta_24h_1,RNA-seq,read1,read2,out
  abeta_24h_2,RNA-seq,read1,read2,out
  abeta_24h_3,RNA-seq,read1,read2,out
  
  ```
  
  `pipeline_interface.yaml`:
  
  ```
  ...
  pipelines:
    sandro_rna_seq:
      name: sandro_rna_seq
      path: sandro_rna_seq.py  # relative to this 
  pipeline_interface.yaml file
      looper_args: True
      arguments:
        "-S": sample_name
        "--input": sample_name_1_read
        "-O": output_folder
  ...
  
  ```
  
  and `project.config.yaml`:
  
  ```
  ...
  derived_columns: [sample_name_1_read,sample_name_2_read,output_folder]
  data_sources:
    read1: /scratch/nv4e/sandro_looper/data/merged/{sample_name}_r1.fq.gz
    read2: /scratch/nv4e/sandro_looper/data/merged/{sample_name}_r2.fq.gz
    out: /scratch/nv4e/sandro_looper_results/results_pipeline/{sample_name}
  ...
  
  ```
  
  In this particular case the parameter `-O` was formed correctly which I was able to see in `sandro_looper_results/submissions/` `.sub` file, however, it was supplied the second time, apparently from the `output_dir` section of the `project_config.yaml` and was overriding the first `-O` command. So, obviously I tried to delete `output_dir` line but it is a required setting for `Looper`, so it can not run without it.
  
  > 3. I have also tried supplying `-O` command in `project_config.yaml` in the `pipeline_args:` section with no success.
  
  I really do not know what I could do at this point. Any suggestions would be greatly appreciated,
  
  **Tags:** scrnaseq, yaml, looper
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-01-20. Score: 1 | Accepted answer
  
  Looper does not generate any of these per-sample outputs.
  
  Looper does not have any control over where your pipeline puts its outputs.
  
  The problem is in your pipeline, not in looper. Assuming you're using a pypiper pipeline, I suggest reading the docs; for example, the "your first pipeline" tutorial: <http://pypiper.readthedocs.io/en/latest/tutorials-first.html>
  
  For real-life example, see [this pipeline](https://github.com/databio/dnameth_pipelines/blob/3e6abd03185f8e2d5ef1c72849afdf9e5033f269/src/rrbs.py)
  
  `outfolder = os.path.abspath(os.path.join(args.output_parent, args.sample_name))
  pm = pypiper.PipelineManager(name="RRBS", outfolder=outfolder, args=args, version=__version__)`
  
  
  ---
  
  