Answers.Body: <p>Could you give an example of names of input files and output files of <code>normalize.py</code> ?</p>
  
  <p>It seems to me that your case could be similar to the one described about the <code>dynamic</code> output:</p>
  
  <p><a href="https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#dynamic-files" rel="nofollow noreferrer">https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#dynamic-files</a></p>
  
  <p>Here you would probably use it for the output of <code>normalization</code> and for the input of <code>all</code> (with a wildcard different from <code>hdf5_file</code>).</p>
  
  <p>Disclaimer: I haven't had the opportunity to use this feature so far.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-09-11T21:30:44.003Z
Answers.Text: Sample name is `{hdf5_file}_{num}.hdf5`, where `num` is an integer.
Answers.UserDisplayName: Nikita Vlasenko
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-09-07T09:30:14.340Z
Answers.LastActivityDate: 2018-09-07T09:30:14.340Z
Answers.UserDisplayName: bli
Body: <p>I have a folder where the <code>hdf5</code> files inputs are (matrices of <code>scRNAseq</code> raw <code>UMI</code> counts). There is a <code>python</code> script that loads them, splits them into multiple matrices, normalizes them, and saves as <code>csv</code> files. When the name pattern of the output files is known, things are easy, I know how to do that, but when one input (<code>hdf5</code> file) can produce various number of outputs (<code>csv</code> files) I am not sure how to approach the issue. Currently my <code>Snakemake</code> file relevant part looks the following way:</p>
  
  <pre><code>hdf5_files,=glob_wildcards('/main/{hdf5_file}.hdf5') 
  
  rule all:
    input:
  ---&gt;  expand('data_files/normalized/{hdf5_file}.csv', hdf5_file=hdf5_files)
      ...
  
  rule normalization:
    input:
       script = 'python/normalize.py',
       path = '/main/{hdf5_file}.hdf5'
    output:
  ---&gt;  path = 'data_files/normalized/{hdf5_file}.csv'
    shell:
       "python {input.script} -data {input.path} -path {output.path}"
  </code></pre>
  
  <p>Above I highlighted the <code>2</code> lines that I am not sure now how I should write. Any suggestions would be greatly appreciated.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-09-05T19:55:59.717Z
Comments.Text: Have you considered using [flag files](https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#flag-files)? Or do you need the names for downstream processing too and can't use `glob` or specify a directory at that point?
Comments.UserDisplayName: Devon Ryan
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-09-05T21:39:16.980Z
Comments.Text: The `rule normalization` processes, say, `2` `hdf5` files and outputs `10` `csv` ones into `data_files/normalized` folder. Then, I want to use `glob_wildcards` to read in these `10` `csv` files and move on.  Further rules are, of course, fully dependent on the `rule normalization` because it generates the input `csv` files. I am not sure how can I use `flag files`, but will try. Thank you!
Comments.UserDisplayName: Nikita Vlasenko
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-09-05T22:19:17.973Z
Comments.Text: Ah, if you need to process each of the resulting csv files independently downstream then there are no snakemake tricks that I know of that will help (I presume `glob_wildcards()` is run during rule parsing, so the CSV files won't be there yet). I suspect you'll want to run `normalization` from a wrapper script beforehand in that case, since then globbing will be certain to work as desired.
Comments.UserDisplayName: Devon Ryan
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-09-06T02:27:23.050Z
Comments.Text: What do you mean from a `wrapper script`? Outside the pipeline, you mean?
Comments.UserDisplayName: Nikita Vlasenko
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-09-06T07:04:22.620Z
Comments.Text: Yes, exactly. A (probably python) script that runs two Snakefiles when you call it, with appropriate options. Then you have the flexibility you need while maintaining the convenience of snakemake.
Comments.UserDisplayName: Devon Ryan
Tags: <snakemake>
Title: Output file names are not known beforehand
ThreadText: # Output file names are not known beforehand
  
  (Asked by: username_0 on 2018-09-05)
  
  I have a folder where the `hdf5` files inputs are (matrices of `scRNAseq` raw `UMI` counts). There is a `python` script that loads them, splits them into multiple matrices, normalizes them, and saves as `csv` files. When the name pattern of the output files is known, things are easy, I know how to do that, but when one input (`hdf5` file) can produce various number of outputs (`csv` files) I am not sure how to approach the issue. Currently my `Snakemake` file relevant part looks the following way:
  
  ```
  hdf5_files,=glob_wildcards('/main/{hdf5_file}.hdf5') 
  
  rule all:
    input:
  --->  expand('data_files/normalized/{hdf5_file}.csv', hdf5_file=hdf5_files)
      ...
  
  rule normalization:
    input:
       script = 'python/normalize.py',
       path = '/main/{hdf5_file}.hdf5'
    output:
  --->  path = 'data_files/normalized/{hdf5_file}.csv'
    shell:
       "python {input.script} -data {input.path} -path {output.path}"
  
  ```
  
  Above I highlighted the `2` lines that I am not sure now how I should write. Any suggestions would be greatly appreciated.
  
  **Tags:** snakemake
  
  > **Comments:**
  > **username_1**: Have you considered using [flag files](https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#flag-files)? Or do you need the names for downstream processing too and can't use `glob` or specify a directory at that point?  
  > **username_0**: The `rule normalization` processes, say, `2` `hdf5` files and outputs `10` `csv` ones into `data_files/normalized` folder. Then, I want to use `glob_wildcards` to read in these `10` `csv` files and move on.  Further rules are, of course, fully dependent on the `rule normalization` because it generates the input `csv` files. I am not sure how can I use `flag files`, but will try. Thank you!  
  > **username_1**: Ah, if you need to process each of the resulting csv files independently downstream then there are no snakemake tricks that I know of that will help (I presume `glob_wildcards()` is run during rule parsing, so the CSV files won't be there yet). I suspect you'll want to run `normalization` from a wrapper script beforehand in that case, since then globbing will be certain to work as desired.  
  > **username_0**: What do you mean from a `wrapper script`? Outside the pipeline, you mean?  
  > **username_1**: Yes, exactly. A (probably python) script that runs two Snakefiles when you call it, with appropriate options. Then you have the flexibility you need while maintaining the convenience of snakemake.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2018-09-07. Score: 4 | Accepted answer
  
  Could you give an example of names of input files and output files of `normalize.py` ?
  
  It seems to me that your case could be similar to the one described about the `dynamic` output:
  
  <https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#dynamic-files>
  
  Here you would probably use it for the output of `normalization` and for the input of `all` (with a wildcard different from `hdf5_file`).
  
  Disclaimer: I haven't had the opportunity to use this feature so far.
  
  > **Comments:**
  > **username_0**: Sample name is `{hdf5_file}_{num}.hdf5`, where `num` is an integer.  
  
  ---
  
  