Hello, 

I have ~300 genome fasta files from NCBI that have different headers and whitespaces. I want to make the header simple and only retain the accession number and the scaffold identity. How do I go about doing it recursively for all the 300 odd genomes I have. 

Here is an example of 2 of the genome headers I have:

    >CM025345.1 Vespula pensylvanica isolate Volc-1 chromosome 10, whole genome shotgun sequence

    >KK868930.1 Zootermopsis nevadensis unplaced genomic scaffold scaffold14, whole genome shotgun sequence

    >CM000285.3 Tribolium castaneum strain Georgia GA2 linkage group LG10, whole genome shotgun sequence'

I want it changed to:

    >CM025345.1_chromosome_10
    >KK868930.1_scaffold14
    >CM000285.3_LG10

How do I go about doing it?

I am currently using sed individually to change the header, but it is taking forever.


Find:

    ^([A-Z][A-Z][0-9].*?) .*(chromosome) ([0-9]+).*

Replace:

    $1_$2_$3

---

Apologies I think I misunderstood your post the first time around. You dont have 300 lines, you have 300 organisms (each with many lines).

Here, there are two approaches I can think of ... 

**Approach 1: Considering the issue as a bioinformatics problem**

0. Obtain a list of Unique Identifiers exactly corresponding to those found in ALL your genome fasta files using $1 from the regex like the one I had in the original answer above. For instance, CM10030304.1, CM42994.1 on and on and on. 

1. Go to a large database like RefSeq. For every UID in your list above, you want to now pull associated records from nuccore. This may be done, for instance, using [eFetch][1].


2. Using any method of accessing `[nuccore][2]` programmatically that you would like (like eFetch), obtain the records linked to the unique identifier. 


3. Finally, process the output from 2. in such a way that you have a [hashtable][2]-like object in which the keys are the UID, and the values are the linked records. For instance, in `Python3`, a `Dict()` comprehension would work, such that you have something like:


**Key:**

    UID=CM025336.1

**Values:** 

    /bp:1..19704315
    /organism="Vespulapensylvanica"
    /mol_type="genomicDNA"
    /isolate="Volc-1"
    /db_xref="taxon:30213"
    /chromosome="1"
    /sex="male"
    /tissue_type="Wholeadult"
    /country="USA:Volcano,HI"
    /lat_lon="19.43N155.21W"
    /collection_date="2017-08"

4. Finally, simply loop over all lines in all 300 genome fasta files. For instance in `python3`, we could write:


    #usr/bin/python
    import re
    allLines=[l.strip() for l in f for f in fasta.readlines() for fasta in fastas]
    for ls in allLines:
        ls=ls.strip()
        nuccoreUID=ls.re("regular expression that generates $1 in my answer above")
        shrinkingLine=ls
        for nuccoreAttribute in allAttributeDict[nuccoreUID]:
            shrinkingLine=shrinkingLine.replace(nuccoreAttribute, "")  

Let me know if you have further questions, but this should more or less take care of everything. `shrinkingLine` should now be a string containing ONLY the parts of your string that were precisely the hardest for which to define matches, as you note in your original post (i.e., LG10, chromosome some such, scaffold what-have-you, etc.). at this point, your computer should be holding everything in memory that you need to do what you propose in your post easily.

**Approach 2: Information entropy-based approaches.** 

In this case, I agree it would not be a bioinformatics problem, rather it would be a statistical learning problem. As such, answer omitted here.

  [1]: https://dataguide.nlm.nih.gov/edirect/efetch.html
  [2]: https://en.wikipedia.org/wiki/Hash_table