Hi guys, 

I have a specific problem about using awk or sed to split a big file to different files.
The big file is like this format(3 columns):

    C    SRR1_45/1    data...
    U    SRR2_34/2    data...
    U    SRR1_33/2    data...
    C    SRR3_22/1    data...
    ....

I want to extract lines with SRR1 to SRR1.txt, lines with SRR2 to SRR2.txt ...lines with SRRn to SRRn.txt. And the output lines should remove 'SRRi_' symbol. But we don't how many n are there.  

    e.g. SRR1.txt will contain:
    C    45/1    data...
    U    33/2    data...


I know it's easy to write a python or perl script to do it. But is there a shell way to do it? taking the advantages of awk or sed.
Let me add some details:
I have 10 such big files to be extracted. And each has more than 1000M lines. So I need to find a efficient way.
The n is random which is not from sequential array.

Thanks!
Tao

Here is a simple way to do it without sorting and with `awk`:

    $ awk '{ split($2, a, "_"); print $1"\t"a[2]"\t"$3 >> a[1]".txt"; }' foo.txt

The file `foo.txt` is a three-column tab-delimited text file containing your data.

Using the `>>` operator appends a line to whatever `SRR*.txt` file exists. Therefore, if you re-run this one-liner, you must first delete any previously-made `SRR*.txt` files, or you will get duplicate lines.

This should be pretty fast, as you're not sorting on IDs. It would be faster, probably, to use a Perl-based approach that opens a pool of file handles, but this should work fine.

Further, if you don't care about the order of lines in the split files, you could use [GNU Parallel][1] with this one-liner to split multiple files `foo1.txt`, `foo2.txt`, etc. simultaneously. Doing the work in parallel may hit a file I/O bottleneck but could give you an overall speed boost, if you use SSDs or other fast storage.


  [1]: https://www.biostars.org/p/63816/