Body: <p>[![example csv file][1]][1]</p>
  <p>First I want to start by parsing and doing some operation on a text/csv file. How could I read a csv/text file in nextflow?</p>
  <p>Suppose, I have above csv file with 3 columns. Lets say I want to create a channel which emits an array from minimum of column 2 (7.9e5) to maximum of column 2 (8.8e5) with an increment of 1e4. How could I create the following channel and later access it in the script? Any help? The</p>
  <pre><code>array=([1]=&quot;7.9e5 8.0e5&quot; [2]=&quot;8.0e5 8.1e5&quot; [3]=&quot;8.1e5 8.2e5&quot; [4]=&quot;8.2e5 8.3e5&quot; [5]=&quot;8.3e5 8.4e5&quot; [6]=&quot;8.4e5 8.5e5&quot; [7]=&quot;8.5e5 8.6e5&quot; [8]=&quot;8.6e5 8.7e5&quot; [9]=&quot;8.7e5 8.8e5&quot; [10]=&quot;8.8e5 8.9e5&quot;)
  </code></pre>
  <p>I need to do some operations with this array in the script which will generate multiple files for each element in the array. These multiple files need to be merged in a single file.
  Sorry, I could not provide any nextflow script that I tried, because I not sure how create an array with <code>splitCsv</code></p>
  <p>I have tried in python:</p>
  <pre><code>df1=pd.read_csv('data1/omni-ec-eur-fin-pass-qc.bim',sep='\t',header=None)
  arr1=np.arange(df1.iloc[:,3].min()-1,df1.iloc[:,3].max()+1,3000000)
  diffs = []
  for x in zip(arr1[0::],arr1[1::]):
      diffs.append(x)
  arr2=np.arange(1,len(diffs)+1,1)
  dict(zip(arr2,diffs))
  </code></pre>
  <p>Which gave me:</p>
  <pre><code>{1: (12343, 3012343),
   2: (3012343, 6012343),
   3: (6012343, 9012343),
   4: (9012343, 12012343),...
  </code></pre>
  <p>But I need something like this:</p>
  <pre><code>([1]=&quot;12343 3012343&quot; [2]=&quot;3012343 6012343&quot; [3]=&quot;6012343   9012343&quot; [4]=&quot;9012343 12012343&quot;....)
  </code></pre>
  <p>It could be arrays (22 arrays for 22 chromosome) or a csv file containing arrays for each chromosome which could be used as an input channel. There are 22 bim files (one for each chromosome) stored in the output channel which would be used to create the above mentioned array.
  The original data inside the bim looks like this:</p>
  <pre><code>    0   1   2   3   4   5
  0   1   rs11240777  0   798959  A   G
  1   1   rs4475691   0   846808  A   G
  2   1   rs7537756   0   854250  G   A
  3   1   rs13302982  0   861808  A   G
  4   1   rs1110052   0   873558  C   A
  </code></pre>
  <p>The 4th column would be used to create the arrays.
  [1]: <a href="https://i.sstatic.net/cxqyl.png" rel="nofollow noreferrer">https://i.sstatic.net/cxqyl.png</a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-09T13:00:32.177Z
Comments.Text: PLease [edit] your question and give us an example file and minimal nextflow script to work with.
Comments.UserDisplayName: terdon
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-10T20:22:21.050Z
Comments.Text: Is there any book to learn nextflow operations? I was trying to find any book to learn nextflow but failed. At least if there is any example of creating array in nextflow would be great.
Comments.UserDisplayName: zillur rahman
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-11T00:08:05.720Z
Comments.Text: Have you read the nextflow documentation? I don't know much about it, but the whole idea is to use it for control flow; it runs other programs and they can have arrays. But next flow isn't a programing language per se. You write your script in Python or peel or whatever and then run the pipeline if scripts with nextflow.
Comments.UserDisplayName: terdon
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-11T10:04:32.877Z
Comments.Text: @zillurrahman Do you really need an array? Or do you mean a channel? Your last paragraph suggests the latter would be a better fit. Please edit your question to clarify exactly what is needed.
Comments.UserDisplayName: Steve
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-11T10:05:58.497Z
Comments.Text: @zillurrahman Generally, it's better to off-load all work to one or more processes. Writing a process to create a range of values, like in your example, is trivial in Perl/Python. You can use your favorite language in a process' script block. Please see: https://www.nextflow.io/docs/latest/process.html#scripts-a-la-carte
Comments.UserDisplayName: Steve
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-11T10:11:34.923Z
Comments.Text: @terdon It's an extension of the Groovy programming language :-)
Comments.UserDisplayName: Steve
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-11T12:02:55.620Z
Comments.Text: @steve sure, but it isn't designed to be used as a regular programing language but as a process control system to build and manage pipelines. It seems far more reasonable to manage the data in another script and then have nextflow manage that script instead of trying to use nextflow as your scripting language directly. Which is exacy what you suggested too.
Comments.UserDisplayName: terdon
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-11T12:08:48.280Z
Comments.Text: Thanks @Steve and @terdon, I edited the above question. Hope, it would clarify my necessity.  However, I am also trying to create such array in `bash` . But, I think probably a nextflow channel would be more appropriate in this case.
Comments.UserDisplayName: zillur rahman
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-11T15:22:16.280Z
Comments.Text: @terdon wasn't disagreeing with you on that...
Comments.UserDisplayName: Steve
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-11T15:39:41.857Z
Comments.Text: @zillurrahman Is there some reason for choosing Bash over Python for example? Bash isn't a good choice given your requirements.
Comments.UserDisplayName: Steve
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-11T16:39:44.140Z
Comments.Text: `@Steve` No reason to choose bash. I tried in python and got an array. `np.arange(df1.iloc[:,3].min()-1,df1.iloc[:,3].max()+1,3000000)` which gives me: `array([    12343,   3012343,   6012343,   9012343,  12012343, ....` . How could I get something like: `([1]="12343 3012343" [2]="3012343 6012343" [3]="6012343   9012343" [4]="9012343 12012343"....)`
Comments.UserDisplayName: zillur rahman
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-11T20:30:35.480Z
Comments.Text: So far, I have done: `diffs = []
  for x in zip(arr1[0::],arr1[1::]):
      diffs.append(x)
  diffs=np.array(diffs)
  arr2=np.arange(1,len(diffs)+1,1)
  arr3=np.column_stack((arr2,diffs))
  arr4=arr3.tolist()` which gave me `[[1, 12343, 3012343],
   [2, 3012343, 6012343],
   [3, 6012343, 9012343],` . Should I convert it to string and use regex to get `([1]="12343 3012343" [2]="3012343 6012343" [3]="6012343   9012343" [4]="9012343 12012343"....)` or there are better alternatives?
Comments.UserDisplayName: zillur rahman
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-11T23:19:01.390Z
Comments.Text: @zillurrahman Do you really need to create the array string? If all you eventually want to do is create a channel of tuples, your first process should write a simple CSV which can then be split using the [splitCsv operator](https://www.nextflow.io/docs/latest/operator.html#splitcsv).
Comments.UserDisplayName: Steve
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-11T23:30:24.850Z
Comments.Text: @zillurrahman Also, your comments above should be edits to your question. Small snippets of unformatted code isn't especially helpful. Please include a small minimal example and the code that reproduces the issue.
Comments.UserDisplayName: Steve
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-12T01:24:39.037Z
Comments.Text: Thanks @`Steve`. I have edited my above question. Hope it makes sense now.
Comments.UserDisplayName: zillur rahman
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-12T02:44:45.310Z
Comments.Text: @zillurrahman Like terdon mentioned earlier, please include some example input (i.e. `cat data1/omni-ec-eur-fin-pass-qc.bim`). Please format it in a code block.
Comments.UserDisplayName: Steve
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-12T03:03:03.050Z
Comments.Text: `@Steve` . The first few rows of the input data is added to the above question.
Comments.UserDisplayName: zillur rahman
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-14T09:16:21.150Z
Comments.Text: Hi, any help/suggestion for the above problem? Did I clarify enough the issue?
Comments.UserDisplayName: zillur rahman
Tags: <nextflow>
Title: parse csv/text file to create a channel with array in nextflow
ThreadText: # parse csv/text file to create a channel with array in nextflow
  
  (Asked by: username_0 on 2021-04-08)
  
  [![example csv file][1]][1]
  
  First I want to start by parsing and doing some operation on a text/csv file. How could I read a csv/text file in nextflow?
  
  Suppose, I have above csv file with 3 columns. Lets say I want to create a channel which emits an array from minimum of column 2 (7.9e5) to maximum of column 2 (8.8e5) with an increment of 1e4. How could I create the following channel and later access it in the script? Any help? The
  
  ```
  array=([1]="7.9e5 8.0e5" [2]="8.0e5 8.1e5" [3]="8.1e5 8.2e5" [4]="8.2e5 8.3e5" [5]="8.3e5 8.4e5" [6]="8.4e5 8.5e5" [7]="8.5e5 8.6e5" [8]="8.6e5 8.7e5" [9]="8.7e5 8.8e5" [10]="8.8e5 8.9e5")
  
  ```
  
  I need to do some operations with this array in the script which will generate multiple files for each element in the array. These multiple files need to be merged in a single file.
  Sorry, I could not provide any nextflow script that I tried, because I not sure how create an array with `splitCsv`
  
  I have tried in python:
  
  ```
  df1=pd.read_csv('data1/omni-ec-eur-fin-pass-qc.bim',sep='\t',header=None)
  arr1=np.arange(df1.iloc[:,3].min()-1,df1.iloc[:,3].max()+1,3000000)
  diffs = []
  for x in zip(arr1[0::],arr1[1::]):
      diffs.append(x)
  arr2=np.arange(1,len(diffs)+1,1)
  dict(zip(arr2,diffs))
  
  ```
  
  Which gave me:
  
  ```
  {1: (12343, 3012343),
   2: (3012343, 6012343),
   3: (6012343, 9012343),
   4: (9012343, 12012343),...
  
  ```
  
  But I need something like this:
  
  ```
  ([1]="12343 3012343" [2]="3012343 6012343" [3]="6012343   9012343" [4]="9012343 12012343"....)
  
  ```
  
  It could be arrays (22 arrays for 22 chromosome) or a csv file containing arrays for each chromosome which could be used as an input channel. There are 22 bim files (one for each chromosome) stored in the output channel which would be used to create the above mentioned array.
  The original data inside the bim looks like this:
  
  ```
      0   1   2   3   4   5
  0   1   rs11240777  0   798959  A   G
  1   1   rs4475691   0   846808  A   G
  2   1   rs7537756   0   854250  G   A
  3   1   rs13302982  0   861808  A   G
  4   1   rs1110052   0   873558  C   A
  
  ```
  
  The 4th column would be used to create the arrays.
  [1]: <https://i.sstatic.net/cxqyl.png>
  
  **Tags:** nextflow
  
  > **Comments:**
  > **username_1**: PLease [edit] your question and give us an example file and minimal nextflow script to work with.  
  > **username_0**: Is there any book to learn nextflow operations? I was trying to find any book to learn nextflow but failed. At least if there is any example of creating array in nextflow would be great.  
  > **username_1**: Have you read the nextflow documentation? I don't know much about it, but the whole idea is to use it for control flow; it runs other programs and they can have arrays. But next flow isn't a programing language per se. You write your script in Python or peel or whatever and then run the pipeline if scripts with nextflow.  
  > **username_2**: @username_0 Do you really need an array? Or do you mean a channel? Your last paragraph suggests the latter would be a better fit. Please edit your question to clarify exactly what is needed.  
  > **username_2**: @username_0 Generally, it's better to off-load all work to one or more processes. Writing a process to create a range of values, like in your example, is trivial in Perl/Python. You can use your favorite language in a process' script block. Please see: https://www.nextflow.io/docs/latest/process.html#scripts-a-la-carte  
  > **username_2**: @username_1 It's an extension of the Groovy programming language :-)  
  > **username_1**: @username_2 sure, but it isn't designed to be used as a regular programing language but as a process control system to build and manage pipelines. It seems far more reasonable to manage the data in another script and then have nextflow manage that script instead of trying to use nextflow as your scripting language directly. Which is exacy what you suggested too.  
  > **username_0**: Thanks @username_2 and @username_1, I edited the above question. Hope, it would clarify my necessity.  However, I am also trying to create such array in `bash` . But, I think probably a nextflow channel would be more appropriate in this case.  
  > **username_2**: @username_1 wasn't disagreeing with you on that...  
  > **username_2**: @username_0 Is there some reason for choosing Bash over Python for example? Bash isn't a good choice given your requirements.  
  > **username_0**: `@username_2` No reason to choose bash. I tried in python and got an array. `np.arange(df1.iloc[:,3].min()-1,df1.iloc[:,3].max()+1,3000000)` which gives me: `array([    12343,   3012343,   6012343,   9012343,  12012343, ....` . How could I get something like: `([1]="12343 3012343" [2]="3012343 6012343" [3]="6012343   9012343" [4]="9012343 12012343"....)`  
  > **username_0**: So far, I have done: `diffs = [] for x in zip(arr1[0::],arr1[1::]):     diffs.append(x) diffs=np.array(diffs) arr2=np.arange(1,len(diffs)+1,1) arr3=np.column_stack((arr2,diffs)) arr4=arr3.tolist()` which gave me `[[1, 12343, 3012343],  [2, 3012343, 6012343],  [3, 6012343, 9012343],` . Should I convert it to string and use regex to get `([1]="12343 3012343" [2]="3012343 6012343" [3]="6012343   9012343" [4]="9012343 12012343"....)` or there are better alternatives?  
  > **username_2**: @username_0 Do you really need to create the array string? If all you eventually want to do is create a channel of tuples, your first process should write a simple CSV which can then be split using the [splitCsv operator](https://www.nextflow.io/docs/latest/operator.html#splitcsv).  
  > **username_2**: @username_0 Also, your comments above should be edits to your question. Small snippets of unformatted code isn't especially helpful. Please include a small minimal example and the code that reproduces the issue.  
  > **username_0**: Thanks @`username_2`. I have edited my above question. Hope it makes sense now.  
  > **username_2**: @username_0 Like username_1 mentioned earlier, please include some example input (i.e. `cat data1/omni-ec-eur-fin-pass-qc.bim`). Please format it in a code block.  
  > **username_0**: `@username_2` . The first few rows of the input data is added to the above question.  
  > **username_0**: Hi, any help/suggestion for the above problem? Did I clarify enough the issue?  
  
  ---
  
  ## 0 answers
  
  