Hello, everyone! I have a file containing huge number of lines and each line have several fields. I sorted the lines based on the first field. Lines with the same first field were grouped together and numbers in each group is flexible. I would like to loop over the whole file and do something on each group of the line separately. I do not know how to loop over the file and make it possible for me to treat the lines group by group. Does anyone could give me some suggestions? Thanks!

Here is a example:

```
seq_1  chr1 12
seq_1  chr2 34
.
.
seq_1  chr3 57
seq_3  chr1 34
seq_3  chr1 26
.
.
seq_3  chr4 47
seq_4  chr9 78
seq_5  chr8 90
seq_5  chr7 77
```

I want to do something on group `seq_1`, `seq_2`, `seq_3`, `seq_4`, `seq_5`, ... separately.
<p>Others have said you can do this with programming logic and Pierre has shown how GNU Parallel could be used to process the blocks. I&#39;d like to expand on the idea of using GNU Parallel to process your blocks and do away with reading/writing the blocks to intermediary files. This way you can process your blocks in parallel and do away with disk IO for an already large file.</p>

<p>Lets create your mock input file:</p>

<pre>
$ echo -e &quot;seq_1\tchr1\t12
seq_1\tchr2\t34
seq_1\tchr3\t57
seq_3\tchr1\t34
seq_3\tchr1\t26
seq_3\tchr4\t47
seq_4\tchr9\t78
seq_5\tchr8\t90
seq_5\tchr7\t77&quot; &gt; input.txt</pre>

<p>Now lets look at inserting a record separator at the start of each block - defined here by whenever the first column changes its value. We&#39;ll use awk to do this and use &quot;----&quot; as the record separator as we don&#39;t expect to find this within our file:</p>

<pre>
$ awk &#39;$1 != previous{print &quot;----&quot;}{previous=$1}1&#39; input.txt
----
seq_1    chr1    12
seq_1    chr2    34
seq_1    chr3    57
----
seq_3    chr1    34
seq_3    chr1    26
seq_3    chr4    47
----
seq_4    chr9    78
----
seq_5    chr8    90
seq_5    chr7    77</pre>

<p>OK, we can now use GNU parallel to process our blocks and using ---- as the record start. We need to remove this line from each block being processed by GNU Parallel - we&#39;ll use --remove-rec-sep<strong> </strong>to achieve this:</p>

<pre>
$ awk &#39;$1 != previous{print &quot;----&quot;}{previous=$1}1&#39; input.txt \
  | parallel --gnu --keep-order --spreadstdin -N 1 --recstart &#39;----\n&#39; --remove-rec-sep cat
seq_1    chr1    12
seq_1    chr2    34
seq_1    chr3    57
seq_3    chr1    34
seq_3    chr1    26
seq_3    chr4    47
seq_4    chr9    78
seq_5    chr8    90
seq_5    chr7    77</pre>

<p>Now we can start put in the my_cmd you want to run on each block using something like:</p>

<pre>
$ awk &#39;$1 != previous{print &quot;----&quot;}{previous=$1}1&#39; input.txt \
  | parallel --gnu --keep-order --spreadstdin -N 1 --recstart &#39;----\n&#39; --remove-rec-sep | my_cmd&#39;</pre>

<p>To test that GNU Parallel is processing each block separately, use awk to prefix each line with the number of lines encountered by each job:</p>

<pre>
$ awk &#39;$1 != previous{print &quot;----&quot;}{previous=$1}1&#39; input.txt \
  | parallel --gnu --keep-order --spreadstdin -N 1 --recstart &#39;----\n&#39; --remove-rec-sep | awk &quot;{print FNR,\$0}&quot;&#39;
1 seq_1    chr1    12
2 seq_1    chr2    34
3 seq_1    chr3    57
1 seq_3    chr1    34
2 seq_3    chr1    26
3 seq_3    chr4    47
1 seq_4    chr9    78
1 seq_5    chr8    90
2 seq_5    chr7    77</pre>

<p>Some advice on using pipelines within GNU Parallel:</p>

<ul>
	<li>Each GNU Parallel job will use more than 1 core. In the above and 1 core for sed - reduce the number of jobs run in parallel to account for this.</li>
	<li>Using pipelines with single and double quotes can be a nightmare - consider moving it into a single function which you cann from GNU Parallel.</li>
</ul>
