Answers.Body: <p>The following workaround seems to work but ideally I don’t want to have to change the <code>script</code> portion of my process so that it works with Docker.</p>
  
  <pre><code>params.ref = 'data/reference/GRCh38.fa'
  
  ref = file(params.ref)
  ref_index_dir = ref.parent
  fastq_channel = Channel.fromFilePairs('data/fastq/*_R{1,2}.fastq.gz')
  
  process map_reads {
      input:
      set sample, '?.fastq.gz' from fastq_channel
      file ref
      file ref_index_dir
  
      output:
      file("${sample}.sam") into sam_channel
  
      script:
      """
      echo 'bwa mem -Y "reference/<span class="math-container">$ref" 1.fastq.gz 2.fastq.gz &gt;$</span>{sample}.sam'
      stat -t reference/<span class="math-container">${ref}.bwt &amp;&amp; touch $</span>{sample}.sam
      """
  }
  </code></pre>
  
  <p>Here, <code>reference</code> inside the script section refers to the parent directory name of the reference, and needs to be adjusted accordingly. Having to refer to <code>reference/$ref</code> is annoying but at least it works — both inside a Docker container and outside of it.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-06T12:38:55.310Z
Answers.LastActivityDate: 2019-06-06T12:38:55.310Z
Answers.UserDisplayName: Konrad Rudolph
Answers.Body: <p>Process input files need to be declared properly to allow nextflow to create the required docker mounts accordingly. Therefore the <code>reference</code> file should be declared in the process input block, eh: </p>
  
  <pre><code>input:
   set sample, file(fastq) from fastq_channel
   file reference 
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-10T11:08:24.530Z
Answers.Text: The issue is that BWA for instance doesn’t just use the “reference” file, otherwise there’d be no issue. But it additionally derives various other files from the reference filename opaquely (i.e. these names are in principle implementation details and not known, except that they all have the reference filename as a prefix).
Answers.UserDisplayName: Konrad Rudolph
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-07T10:08:32.027Z
Answers.LastActivityDate: 2019-06-07T10:08:32.027Z
Answers.UserDisplayName: pditommaso
Body: <p>I’m trying to run a Nextflow workflow in a custom Docker container. Without Docker, the workflow succeeds. But running it inside the container leads to an error because a dependent file cannot be found.</p>
  
  <p>Here’s a minimal script to illustrate the issue (similar to the <a href="https://www.nextflow.io/docs/latest/basic.html" rel="nofollow noreferrer">basic Nextflow script example</a>):</p>
  
  <pre><code>params.reference = "/data/reference/GRCh38.fa"
  
  reference = file(params.reference)
  fastq_channel = Channel.fromFilePairs('/data/fastq/*_R{1,2}.fastq.gz')
  
  process map_reads {
      input:
      set sample, '?.fastq.gz' from fastq_channel
  
      output:
      file("${sample}.sam") into sam_channel
  
      script:
      """
      echo 'bwa mem -Y "<span class="math-container">$reference" 1.fastq.gz 2.fastq.gz &gt;$</span>{sample}.sam'
      stat -t "${reference}.bwt"
      """
  }
  </code></pre>
  
  <p>Running this inside a container will cause the path of the query, <code>/some/fastq/</code>, to be mounted inside the container. However, the path to the database, <code>/data/reference</code>, will <em>not</em> be mounted. Consequently, I get the following execution error:</p>
  
  <blockquote>
    <p>stat: cannot stat 'GRCh38-chr21.fa.bwt': No such file or directory</p>
  </blockquote>
  
  <p>Without the container, the command succeeds.</p>
  
  <hr>
  
  <p>I understand <em>why</em> the error occurs: Nextflow doesn’t track the dependency on <code>$reference</code> inside the command, and thus doesn’t mount its path in the container.</p>
  
  <p>My question is, <strong>how do I fix this?</strong> The documentation is silent on this regard, and none of the workflows I can find online address the problem. For example, the <a href="https://github.com/IARCbioinfo" rel="nofollow noreferrer">GATK4 IARC workflows</a> would presumably have the same issue, and aren’t configured for use with Docker.</p>
  
  <p>Just adding the <code>reference</code> as a process input is insufficient, since BWA requires various other index files. Am I supposed to add these all manually as inputs?</p>
  
Tags: <docker><nextflow>
Title: Run Nextflow with file dependencies inside a Docker container?
ThreadText: # Run Nextflow with file dependencies inside a Docker container?
  
  (Asked by: username_0 on 2019-06-06)
  
  I’m trying to run a Nextflow workflow in a custom Docker container. Without Docker, the workflow succeeds. But running it inside the container leads to an error because a dependent file cannot be found.
  
  Here’s a minimal script to illustrate the issue (similar to the [basic Nextflow script example](https://www.nextflow.io/docs/latest/basic.html)):
  
  ```
  params.reference = "/data/reference/GRCh38.fa"
  
  reference = file(params.reference)
  fastq_channel = Channel.fromFilePairs('/data/fastq/*_R{1,2}.fastq.gz')
  
  process map_reads {
      input:
      set sample, '?.fastq.gz' from fastq_channel
  
      output:
      file("${sample}.sam") into sam_channel
  
      script:
      """
      echo 'bwa mem -Y "$reference" 1.fastq.gz 2.fastq.gz >${sample}.sam'
      stat -t "${reference}.bwt"
      """
  }
  
  ```
  
  Running this inside a container will cause the path of the query, `/some/fastq/`, to be mounted inside the container. However, the path to the database, `/data/reference`, will *not* be mounted. Consequently, I get the following execution error:
  
  > stat: cannot stat 'GRCh38-chr21.fa.bwt': No such file or directory
  
  Without the container, the command succeeds.
  
  ---
  
  I understand *why* the error occurs: Nextflow doesn’t track the dependency on `$reference` inside the command, and thus doesn’t mount its path in the container.
  
  My question is, **how do I fix this?** The documentation is silent on this regard, and none of the workflows I can find online address the problem. For example, the [GATK4 IARC workflows](https://github.com/IARCbioinfo) would presumably have the same issue, and aren’t configured for use with Docker.
  
  Just adding the `reference` as a process input is insufficient, since BWA requires various other index files. Am I supposed to add these all manually as inputs?
  
  **Tags:** docker, nextflow
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_0 on 2019-06-06. Score: 3
  
  The following workaround seems to work but ideally I don’t want to have to change the `script` portion of my process so that it works with Docker.
  
  ```
  params.ref = 'data/reference/GRCh38.fa'
  
  ref = file(params.ref)
  ref_index_dir = ref.parent
  fastq_channel = Channel.fromFilePairs('data/fastq/*_R{1,2}.fastq.gz')
  
  process map_reads {
      input:
      set sample, '?.fastq.gz' from fastq_channel
      file ref
      file ref_index_dir
  
      output:
      file("${sample}.sam") into sam_channel
  
      script:
      """
      echo 'bwa mem -Y "reference/$ref" 1.fastq.gz 2.fastq.gz >${sample}.sam'
      stat -t reference/${ref}.bwt && touch ${sample}.sam
      """
  }
  
  ```
  
  Here, `reference` inside the script section refers to the parent directory name of the reference, and needs to be adjusted accordingly. Having to refer to `reference/$ref` is annoying but at least it works — both inside a Docker container and outside of it.
  
  
  ---
  
  ### Answer by username_1 on 2019-06-07. Score: 1
  
  Process input files need to be declared properly to allow nextflow to create the required docker mounts accordingly. Therefore the `reference` file should be declared in the process input block, eh:
  
  ```
  input:
   set sample, file(fastq) from fastq_channel
   file reference 
  
  ```
  
  > **Comments:**
  > **username_0**: The issue is that BWA for instance doesn’t just use the “reference” file, otherwise there’d be no issue. But it additionally derives various other files from the reference filename opaquely (i.e. these names are in principle implementation details and not known, except that they all have the reference filename as a prefix).  
  
  ---
  
  