Answers.Body: <p>If you are using <code>bash</code>, you can enable the <code>globstar</code> option (<code>shopt -s globstar</code>) which allows <code>**</code> to match recursively. That way, <code>**/*fasta</code> will find <em>all</em> files whose name ends with <code>fastq</code> in this directory and all of its subdirectories. So that will let you easily iterate over all the target names. Next, if you have the perl <code>rename</code> command (called <code>rename</code> on Debian and Ubuntu and similar systems, <code>prename</code> or <code>perl-rename</code> on others), you can do it quite easily.</p>
  <p>Note that <code>perl-rename</code> has the <code>-n</code> option that makes it print out the renaming it would do without actually renaming anything. So I suggest you first run this to test what you will be doing:</p>
  <pre><code>shopt -s globstar
  for file in **/*.fasta; do
     echo perl -i.bak -pe 's/^\s*&gt;.*uce:(uce-\d+).*(\.unaligned.fasta)/$1$2/' &quot;$file&quot;
     rename -n 's/^.*uce:(uce-\d+).*(\.unaligned.fasta)/$1$2/' &quot;$file&quot;
  done
  </code></pre>
  <p>That will <code>echo</code> the perl command, so you can try it manually first and make sure it works, and will print out the <code>rename</code> operations without actually doing anything. If you are satisfied it will work, run again without the <code>echo</code> or <code>-n</code>:</p>
  <pre><code>for file in **/*fasta; do
      perl -i.bak -pe 's/^\s*&gt;.*uce:(uce-\d+).*(\.unaligned.fasta)/$1$2/' &quot;$file
      rename 's/^.*uce:(uce-\d+).*(\.unaligned.fasta)/$1$2/' &quot;$file&quot;
  done
  </code></pre>
  <p>Or, if you don't have <code>perl-rename</code>, you can use the shell's string manipulation for this:</p>
  <pre><code>for file in **/*.fasta; do
      perl -i.bak -pe 's/^\s*&gt;.*uce:(uce-\d+).*(\.unaligned.fasta)/$1$2/' &quot;$file
      tmpFileName=&quot;${file#*uce:}&quot;
      mv -- &quot;$file&quot; &quot;${tmpFileName/|*/.unaligned.fasta}&quot;
  done
  </code></pre>
  <h3>Explanation</h3>
  <ul>
  <li><p><code>shopt -s globstar</code>: this enables the <code>globstar</code> bash option for the current shell session. It only needs to be run once, its effects will last until you close the terminal or start a new shell session. As for what it does, see <code>man bash</code>:</p>
  <blockquote>
  <p>globstar</p>
  <p>If set, the pattern <code>**</code> used in a pathname expansion context will match all files and zero or more directories and subdirectories. If the pattern is followed by a <code>/</code>, only directories and subdirectories match.</p>
  </blockquote>
  </li>
  <li><p><code>for file in **/*fasta; do ... ; done</code>: iterate over all files (and directories) whose name ends in <code>.fasta</code>, saving each file name as <code>$file</code>.</p>
  </li>
  <li><p><code>perl -i.bak -pe 's/^\s*&gt;.*(uce-\d+).*(\.unaligned.fasta)/$1$2/' &quot;$file</code>: this changes the fasta header. The <code>-i.bak</code> option tells <code>perl</code> to edit the file in place, but to save the original file with a <code>.bak</code> extension. This can help you revert any changes if there is a mistake. Once you have finished, you could run <code>rm **/*fasta.bak</code> to delete the backups. Alternatively, just remove the <code>-i.bak</code> if you don't want to keep backups.</p>
  <p>The <code>-p</code> option tells <code>perl</code> to process its input line by line and print every line after applying whatever script you give it with <code>-e</code>.</p>
  <p>The <code>s/old/new/</code> is the substitution operator, same as <code>sed</code>. The regular expression is looking for 0 or more whitespace characters and then a <code>&gt;</code> at the beginning of the line (<code>^\s*&gt;</code>). It then allows for 0 or more characters (<code>.*</code>) until it finds the string <code>uce:</code> followed by <code>uce-</code> and one or more digits (<code>uce:(uce-\d+)</code>). The parentheses &quot;capture&quot; the pattern, storing it as <code>$1</code>. Next, we want 0 or more characters again until we reach <code>.unaligned.fasta</code> (<code>.*(\.unaligned.fasta)</code>). We capture the <code>.unaligned.fasta</code> as <code>$2</code> (second set of parentheses) only to avoid typing it out again. Note that this assumes you will never have anything after the <code>.unaligned.fasta</code>. Finally, we replace everything matched above with just <code>$1</code> (the <code>uce-numbers</code>) and <code>$2</code> (<code>.unaligned.fasta</code>).</p>
  </li>
  <li><p><code>rename 's/^.*uce:(uce-\d+).*(\.unaligned.fasta)/$1$2/' &quot;$file&quot;</code>: the rename opration. This will match everything from the beginning of the file name until the last occurrence of <code>uce:uce-NNNN</code>, capture the <code>uce-NNNN</code> as <code>$1</code> then skip the remaining characters until <code>.unaligned.fasta</code> and finally rename as just <code>$1$2</code> (the capturing logic is the same as above).</p>
  </li>
  <li><p><code>tmpFileName=&quot;${file#*uce:}&quot;</code>: The <code>${var#pattern}</code> syntax means &quot;print the value of <code>$var</code>, but remove the shortest match of <code>pattern</code> from the front of the variable's value&quot;. The <code>${var//pattern/replacement}</code> syntax means &quot;print the value of <code>$var</code> and replace the first occurrence of <code>pattern</code> with <code>replacement</code>. Here, the pattern is <code>|*</code> which means &quot;anything after a <code>|</code>&quot; (note that this is shell stuff and is using globs and not regular expressions). This is probably easier to understand with an example:</p>
  <pre><code>$ echo &quot;$file&quot;
  947-length-1150-cov-1000|contig:JAECWU010000213.1|slice:817050-818200|uce:uce-1452|match:817550-817700|orient:{'+'}|probes:4.unaligned.fasta
  $ tmpFileName=&quot;${file#*uce:}&quot;
  $ echo &quot;$tmpFileName&quot;
  uce-1452|match:817550-817700|orient:{'+'}|probes:4.unaligned.fasta
  $ echo &quot;${tmpFileName/|*/.unaligned.fasta}&quot;
  uce-1452.unaligned.fasta
  </code></pre>
  <p>So, given <code>tmpFileName=&quot;${file#*uce:}&quot;</code>, <code>&quot;${tmpFileName/|*/.unaligned.fasta}&quot;</code> is the desired file name and this is what we pass to <code>mv</code>.</p>
  </li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-24T15:21:12.893Z
Answers.LastActivityDate: 2021-05-24T15:33:58.200Z
Answers.LastEditDate: 2021-05-24T15:33:58.200Z
Answers.UserDisplayName: terdon
Answers.Body: <p>I think you're working too hard to capture the <code>uce:</code> expression.</p>
  <p>Try</p>
  <pre><code>sed -Ee '1s/^.+uce:(uce-[0-9]+).+/\1.unaligned.fasta/' file_glob
  </code></pre>
  <p>to replace the first line content. You may want to add the in-place flag after testing the expression on a few files.</p>
  <p>For the <code>mv</code>, use command substitution, which runs the command in a sub-shell and uses the output value as the parameter:</p>
  <pre><code>for f in file_glob
  do    
      mv -- &quot;<span class="math-container">$f" "$</span>(echo \&quot;$f\&quot; | sed '...')&quot;
  done
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-24T15:22:59.443Z
Answers.Text: Note that your `sed` will not do what the OP wants: it will change the first line to `>uce-1452` and not `uce-1452.unaligned.fasta`.
Answers.UserDisplayName: terdon
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-24T15:32:58.180Z
Answers.Text: Thanks, @terdon. I was nudging OP in the right direction. The `.unaligned.fasta` is trivial to add once OP switches to a simpler solving mindset. I'll add it in to make the solution more complete though.
Answers.UserDisplayName: Ram RS
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-24T15:35:52.773Z
Answers.Text: Force of habit: that protects from the (admittedly unlikely) case where a fasta file's name starts with `-`. Without the `--`, mv will treat that as an option and fail. Try  `mv -badName goodName` and contrast with `mv -- -badName goodName`.
Answers.UserDisplayName: terdon
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-24T15:36:48.137Z
Answers.Text: Thanks, I did some googling and saw why the `--` prevents filenames from manifesting nightmares in worst case scenarios, which is why I deleted my earlier comment that asked you why you recommend `mv --`.
Answers.UserDisplayName: Ram RS
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-24T15:37:47.130Z
Answers.Text: Of course, if I were being careful, I should also have added `--` to the `echo` in case the file name begins with `-n`. But I forgot and it really is vanishingly unlikely that this would happen with actual fasta file names.
Answers.UserDisplayName: terdon
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-24T16:06:53.570Z
Answers.Text: @terdon - Only `<()` and `>()` are process substitution. `$()` is not that. It is a different invocation but definitely not process substitution as it does not replace a file. EDIT: I think it's called command substitution. "Subshell" is closer than "process substitution" though, so please edit judiciously.
Answers.UserDisplayName: Ram RS
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-24T16:17:14.573Z
Answers.Text: Argh! You're absolutely right. I meant command substitution. Sorry, I should indeed be more careful. I just wanted to remove the "subshell" since that is misleading.
Answers.UserDisplayName: terdon
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-24T16:37:38.917Z
Answers.Text: Let us [continue this discussion in chat](https://chat.stackexchange.com/rooms/124629/discussion-between-terdon-and-ram-rs).
Answers.UserDisplayName: terdon
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-25T16:45:33.387Z
Answers.Text: Brilliant, now to try the above. Thank you!
Answers.UserDisplayName: Malia W
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-28T21:54:50.823Z
Answers.Text: Thanks for the suggestions above @terdon. However, both suggestions (if I run from the main directory, take the .fasta files out of the subdirectories, correctly renamed, but moved from their subdirectory. This causes them to overwrite each other, as each subdirectory may have fastas with the same uce-number assigned. I ran: for file in **/*.fasta;do
   perl -i.bak -pe 's/^\s*>.*uce:(uce-\d+).*(\.unaligned.fasta)/$1$2/' "$file"
   rename 's/^\s*>.*uce:(uce-\d+).*(\.unaligned.fasta)/$1$2/' "$file"
  done
Answers.UserDisplayName: Malia W
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-22T04:54:12.940Z
Answers.LastActivityDate: 2021-05-24T16:11:56.703Z
Answers.LastEditDate: 2021-05-24T16:11:56.703Z
Answers.UserDisplayName: Ram RS
Body: <p>I have a <code>MainDirectory/hundreds_of_subdirectories/thousands_of_fastas</code> structure. For each fasta in the bottom level, I am trying to change the file name as well as the first line in the file to retain only some parts of the file.</p>
  <p>So that the following text, which makes up both the file name and the first name of the file:</p>
  <pre><code>947-length-1150-cov-1000|contig:JAECWU010000213.1|slice:817050-818200|uce:uce-1452|match:817550-817700|orient:{'+'}|probes:4.unaligned.fasta
  </code></pre>
  <p>becomes just:</p>
  <pre><code>uce-1452.unaligned.fasta
  </code></pre>
  <p>For editing the first line in the file, I have tried</p>
  <pre><code>sed '1 s/^[^|uce-]*(|uce-[0-9]).*/\1/' hundreds_of_subdirectories/*
  </code></pre>
  <p>but this isn't the ticket..</p>
  <p>Then messing around with some kind of loop...</p>
  <pre><code>for i in *\*.unaligned.fasta
  do
    sed -E 's/^[^|uce-]*(|uce-[0-9]).*/\1/'
  done
  </code></pre>
  <p>For the file name, it was suggested that I use rename instead of sed, but with sed I have tried something like:</p>
  <pre><code>for x in hundreds_of_subdirectories/thousands_of_fastas*unaligned.fasta
  do
    echo $x | sed -r 's/^[^|uce-]*(|uce-[0-9]).*/mv' -v &quot;\0&quot; &quot;|uce-[0-9].*/\1/.unaligned.fasta&quot;/
  done
  </code></pre>
  <p>Am I even barking up the right tree? Thanks!</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-05-24T14:36:19.250Z
Comments.Text: What operating system are you using?
Comments.UserDisplayName: terdon
Tags: <fasta><bash>
Title: Retain only part of a file name and fasta header in fasta directory
ThreadText: # Retain only part of a file name and fasta header in fasta directory
  
  (Asked by: username_0 on 2021-05-21)
  
  I have a `MainDirectory/hundreds_of_subdirectories/thousands_of_fastas` structure. For each fasta in the bottom level, I am trying to change the file name as well as the first line in the file to retain only some parts of the file.
  
  So that the following text, which makes up both the file name and the first name of the file:
  
  ```
  947-length-1150-cov-1000|contig:JAECWU010000213.1|slice:817050-818200|uce:uce-1452|match:817550-817700|orient:{'+'}|probes:4.unaligned.fasta
  
  ```
  
  becomes just:
  
  ```
  uce-1452.unaligned.fasta
  
  ```
  
  For editing the first line in the file, I have tried
  
  ```
  sed '1 s/^[^|uce-]*(|uce-[0-9]).*/\1/' hundreds_of_subdirectories/*
  
  ```
  
  but this isn't the ticket..
  
  Then messing around with some kind of loop...
  
  ```
  for i in *\*.unaligned.fasta
  do
    sed -E 's/^[^|uce-]*(|uce-[0-9]).*/\1/'
  done
  
  ```
  
  For the file name, it was suggested that I use rename instead of sed, but with sed I have tried something like:
  
  ```
  for x in hundreds_of_subdirectories/thousands_of_fastas*unaligned.fasta
  do
    echo $x | sed -r 's/^[^|uce-]*(|uce-[0-9]).*/mv' -v "\0" "|uce-[0-9].*/\1/.unaligned.fasta"/
  done
  
  ```
  
  Am I even barking up the right tree? Thanks!
  
  **Tags:** fasta, bash
  
  > **Comments:**
  > **username_1**: What operating system are you using?  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2021-05-24. Score: 1
  
  If you are using `bash`, you can enable the `globstar` option (`shopt -s globstar`) which allows `**` to match recursively. That way, `**/*fasta` will find *all* files whose name ends with `fastq` in this directory and all of its subdirectories. So that will let you easily iterate over all the target names. Next, if you have the perl `rename` command (called `rename` on Debian and Ubuntu and similar systems, `prename` or `perl-rename` on others), you can do it quite easily.
  
  Note that `perl-rename` has the `-n` option that makes it print out the renaming it would do without actually renaming anything. So I suggest you first run this to test what you will be doing:
  
  ```
  shopt -s globstar
  for file in **/*.fasta; do
     echo perl -i.bak -pe 's/^\s*>.*uce:(uce-\d+).*(\.unaligned.fasta)/$1$2/' "$file"
     rename -n 's/^.*uce:(uce-\d+).*(\.unaligned.fasta)/$1$2/' "$file"
  done
  
  ```
  
  That will `echo` the perl command, so you can try it manually first and make sure it works, and will print out the `rename` operations without actually doing anything. If you are satisfied it will work, run again without the `echo` or `-n`:
  
  ```
  for file in **/*fasta; do
      perl -i.bak -pe 's/^\s*>.*uce:(uce-\d+).*(\.unaligned.fasta)/$1$2/' "$file
      rename 's/^.*uce:(uce-\d+).*(\.unaligned.fasta)/$1$2/' "$file"
  done
  
  ```
  
  Or, if you don't have `perl-rename`, you can use the shell's string manipulation for this:
  
  ```
  for file in **/*.fasta; do
      perl -i.bak -pe 's/^\s*>.*uce:(uce-\d+).*(\.unaligned.fasta)/$1$2/' "$file
      tmpFileName="${file#*uce:}"
      mv -- "$file" "${tmpFileName/|*/.unaligned.fasta}"
  done
  
  ```
  
  ### Explanation
  
  * `shopt -s globstar`: this enables the `globstar` bash option for the current shell session. It only needs to be run once, its effects will last until you close the terminal or start a new shell session. As for what it does, see `man bash`:
  
    > globstar
    >
    > If set, the pattern `**` used in a pathname expansion context will match all files and zero or more directories and subdirectories. If the pattern is followed by a `/`, only directories and subdirectories match.
  * `for file in **/*fasta; do ... ; done`: iterate over all files (and directories) whose name ends in `.fasta`, saving each file name as `$file`.
  * `perl -i.bak -pe 's/^\s*>.*(uce-\d+).*(\.unaligned.fasta)/$1$2/' "$file`: this changes the fasta header. The `-i.bak` option tells `perl` to edit the file in place, but to save the original file with a `.bak` extension. This can help you revert any changes if there is a mistake. Once you have finished, you could run `rm **/*fasta.bak` to delete the backups. Alternatively, just remove the `-i.bak` if you don't want to keep backups.
  
    The `-p` option tells `perl` to process its input line by line and print every line after applying whatever script you give it with `-e`.
  
    The `s/old/new/` is the substitution operator, same as `sed`. The regular expression is looking for 0 or more whitespace characters and then a `>` at the beginning of the line (`^\s*>`). It then allows for 0 or more characters (`.*`) until it finds the string `uce:` followed by `uce-` and one or more digits (`uce:(uce-\d+)`). The parentheses "capture" the pattern, storing it as `$1`. Next, we want 0 or more characters again until we reach `.unaligned.fasta` (`.*(\.unaligned.fasta)`). We capture the `.unaligned.fasta` as `$2` (second set of parentheses) only to avoid typing it out again. Note that this assumes you will never have anything after the `.unaligned.fasta`. Finally, we replace everything matched above with just `$1` (the `uce-numbers`) and `$2` (`.unaligned.fasta`).
  * `rename 's/^.*uce:(uce-\d+).*(\.unaligned.fasta)/$1$2/' "$file"`: the rename opration. This will match everything from the beginning of the file name until the last occurrence of `uce:uce-NNNN`, capture the `uce-NNNN` as `$1` then skip the remaining characters until `.unaligned.fasta` and finally rename as just `$1$2` (the capturing logic is the same as above).
  * `tmpFileName="${file#*uce:}"`: The `${var#pattern}` syntax means "print the value of `$var`, but remove the shortest match of `pattern` from the front of the variable's value". The `${var//pattern/replacement}` syntax means "print the value of `$var` and replace the first occurrence of `pattern` with `replacement`. Here, the pattern is `|*` which means "anything after a `|`" (note that this is shell stuff and is using globs and not regular expressions). This is probably easier to understand with an example:
  
    ```
    $ echo "$file"
    947-length-1150-cov-1000|contig:JAECWU010000213.1|slice:817050-818200|uce:uce-1452|match:817550-817700|orient:{'+'}|probes:4.unaligned.fasta
    $ tmpFileName="${file#*uce:}"
    $ echo "$tmpFileName"
    uce-1452|match:817550-817700|orient:{'+'}|probes:4.unaligned.fasta
    $ echo "${tmpFileName/|*/.unaligned.fasta}"
    uce-1452.unaligned.fasta
  
    ```
  
    So, given `tmpFileName="${file#*uce:}"`, `"${tmpFileName/|*/.unaligned.fasta}"` is the desired file name and this is what we pass to `mv`.
  
  
  ---
  
  ### Answer by username_2 on 2021-05-22. Score: 1
  
  I think you're working too hard to capture the `uce:` expression.
  
  Try
  
  ```
  sed -Ee '1s/^.+uce:(uce-[0-9]+).+/\1.unaligned.fasta/' file_glob
  
  ```
  
  to replace the first line content. You may want to add the in-place flag after testing the expression on a few files.
  
  For the `mv`, use command substitution, which runs the command in a sub-shell and uses the output value as the parameter:
  
  ```
  for f in file_glob
  do    
      mv -- "$f" "$(echo \"$f\" | sed '...')"
  done
  
  ```
  
  > **Comments:**
  > **username_1**: Note that your `sed` will not do what the OP wants: it will change the first line to `>uce-1452` and not `uce-1452.unaligned.fasta`.  
  > **username_2**: Thanks, @username_1. I was nudging OP in the right direction. The `.unaligned.fasta` is trivial to add once OP switches to a simpler solving mindset. I'll add it in to make the solution more complete though.  
  > **username_1**: Force of habit: that protects from the (admittedly unlikely) case where a fasta file's name starts with `-`. Without the `--`, mv will treat that as an option and fail. Try  `mv -badName goodName` and contrast with `mv -- -badName goodName`.  
  > **username_2**: Thanks, I did some googling and saw why the `--` prevents filenames from manifesting nightmares in worst case scenarios, which is why I deleted my earlier comment that asked you why you recommend `mv --`.  
  > **username_1**: Of course, if I were being careful, I should also have added `--` to the `echo` in case the file name begins with `-n`. But I forgot and it really is vanishingly unlikely that this would happen with actual fasta file names.  
  > **username_2**: @username_1 - Only `<()` and `>()` are process substitution. `$()` is not that. It is a different invocation but definitely not process substitution as it does not replace a file. EDIT: I think it's called command substitution. "Subshell" is closer than "process substitution" though, so please edit judiciously.  
  > **username_1**: Argh! You're absolutely right. I meant command substitution. Sorry, I should indeed be more careful. I just wanted to remove the "subshell" since that is misleading.  
  > **username_1**: Let us [continue this discussion in chat](https://chat.stackexchange.com/rooms/124629/discussion-between-username_1-and-username_2).  
  > **username_0**: Brilliant, now to try the above. Thank you!  
  > **username_0**: Thanks for the suggestions above @username_1. However, both suggestions (if I run from the main directory, take the .fasta files out of the subdirectories, correctly renamed, but moved from their subdirectory. This causes them to overwrite each other, as each subdirectory may have fastas with the same uce-number assigned. I ran: for file in **/*.fasta;do  perl -i.bak -pe 's/^\s*>.*uce:(uce-\d+).*(\.unaligned.fasta)/$1$2/' "$file"  rename 's/^\s*>.*uce:(uce-\d+).*(\.unaligned.fasta)/$1$2/' "$file" done  
  
  ---
  
  