Answers.Body: <p>I <em>think</em> you are trying to use <code>awk</code> to get characters 3 through 7 (inclusive) of each input sequence. If so, a working version of your approach would be this:</p>
  
  <pre><code>my <span class="math-container">$splice = `echo "$</span>string" | awk 'BEGIN {FS=\"\"} {x = <span class="math-container">$n; while (x &lt; $</span>z){ printf \"%s\",\$x; ++x} print ""}' 2&gt;1`;
  </code></pre>
  
  <p>That, when added to your perl script, produces:</p>
  
  <pre><code>$ foo.pl
  VLVIL
  VLVIL
  VLVIL
  </code></pre>
  
  <p>That isn't the output you show, but the output you show is just the first 5 letters of each sequence, which doesn't seem to correspond to what your script is doing. </p>
  
  <p>Now, the reason your version didn't work, is that the <code>while</code> was like so:</p>
  
  <pre><code>while (x &lt; $z); print \$ x; ++x}
  </code></pre>
  
  <p>Assuming the <code>\$ x</code> was a typo, the problem is that the <code>++x</code> isn't part of the while loop. So that should never even have exited. Seeing as we're in the same time zone, and given the time, I am guessing you're tired and that you miscopied something when trying to give us a reproducible example. In any case, to get the loop right, you need <code>{}</code>:</p>
  
  <pre><code>while (x &lt; $z){print \$x; ++x}
  </code></pre>
  
  <p>However, if all you want is a substring, a loop is a very inefficient way of doing that. You can instead use <code>awk</code>'s <code>substr</code> command:</p>
  
  <pre><code>  substr(s, m[, n ])
               Return  the at most n-character substring of s that begins at position
               m, numbering from 1. If n is omitted, or if n specifies  more  charac‐
               ters than are left in the string, the length of the substring shall be
               limited by the length of the string s.
  </code></pre>
  
  <p>So, in your script, that would be:</p>
  
  <pre><code>my <span class="math-container">$splice = `echo "$</span>string" | awk '{print substr(\$0,$n,5)}'
  </code></pre>
  
  <hr>
  
  <p>Since you're already using a language that excels in string manipulation, I really don't understand why you would try to use <code>awk</code> instead. It will not be faster or more efficient than the same thing in perl:</p>
  
  <pre><code>print substr(<span class="math-container">$_,$</span>n,5) . "\n" for @array;
  </code></pre>
  
  <p>By calling <code>awk</code> you are also calling the shell which will be slower by definition, especially if you're <code>echo</code>ing thousands of lines. In fact, I tried this with 15000 sequences (just repetitions of your first one) and using either the <code>awk</code> loop I gave above, or the <code>awk</code>+<code>substr</code>, or the native perl <code>substr</code>:</p>
  
  <ol>
  <li>awk loop, average time taken to run the same thing 100 times: 0.0525</li>
  <li>awk substr, average time taken to run the same thing 100 times: 0.0522s</li>
  <li>perl substr, average time taken to run the same thing 100 times: 0.0464</li>
  </ol>
  
  <p>And you can make it even faster by skipping the array and iterating over the keys directly:</p>
  
  <pre><code>print substr(<span class="math-container">$dna{$_},$</span>n,5) . "\n" for keys(%dna);
  </code></pre>
  
  <p>That averaged at 0.0376s over 100 runs.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-25T14:51:08.157Z
Answers.Text: Huge thanks @terdon. You are right I should have performed a runtime check :-( such as use Timer::Runtime. I worked on the implicit assumption awk was faster for this operation. David Cross "lectured" on runtime in detail in "Data Munging for Perl": There is no denying that 0.0525 vs. 0.0376s is 30% faster.
Answers.UserDisplayName: M__
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-25T15:29:51.773Z
Answers.Text: @MichaelG. in my experience, awk is blindingly fast when your data are organized in fields because of its implicit splitting. If not, however, it doesn't give any real advantage over a language like Perl which is simply brilliant for text parsing.
Answers.UserDisplayName: terdon
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-25T01:14:44.037Z
Answers.LastActivityDate: 2019-02-25T01:35:36.460Z
Answers.LastEditDate: 2019-02-25T01:35:36.460Z
Answers.UserDisplayName: terdon
Answers.Body: <p>If you have the sequences in a seperate file like this:</p>
  
  <pre><code>$ cat input
  LGVLVILLMVQEGLKKRMTTKIIISTSMAVLVAMILGGFSMSDLAKLAILMGATFAEMNTGGDVAHLALIAAFKVRPALLVSFIF
  LGVLVILLMVQEGLKKRMTTKIIISTSMAVLVAMILGGFSMSDLAKLAILMGATFAEMNTGGDVXXXXXXAAFKVRPALLVSFIF
  LGVLVILLMVQEGLKKRMTTKIIISTSMAVLVAMILGGFSMSDLAKLAILMGATFAEMNTGGZZZZZZLIAAFKVRPALLVSFIF
  </code></pre>
  
  <p>you can use a simple <code>cut</code> to fetch a substring:</p>
  
  <pre><code>$ cut -c1-5 input
  LGVLV
  LGVLV
  LGVLV
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-25T06:14:56.173Z
Answers.LastActivityDate: 2019-02-25T06:14:56.173Z
Answers.UserDisplayName: finswimmer
Body: <p>I'm looking to splice out specific amino acid alignment positions as a continuous motif from an alignment (~15 000 sequences). The code is written in Perl (I agree not many developers now) but splicing method is awk which a lot of you know. Please note the hash is "tied" (not shown), this just means the order of the sequences is preserved.  I can get the awk code below to return single amino acid positions. </p>
  
  <p>I am struggling to get a loop working in awk.</p>
  
  <p>I agree the Perl wrapper makes it more complicated, but its part of a huge data pipeline (all written in Perl).</p>
  
  <pre class="lang-perl prettyprint-override"><code>#!/usr/local/ActivePerl-5.20/bin/env perl
  #use strict;
  use warnings;
  
  # I loaded the alignment into a hash, where key = sequence and value = protein
  my %dna = (
      virus1 =&gt; "LGVLVILLMVQEGLKKRMTTKIIISTSMAVLVAMILGGFSMSDLAKLAILMGATFAEMNTGGDVAHLALIAAFKVRPALLVSFIF", 
      virus2 =&gt; "LGVLVILLMVQEGLKKRMTTKIIISTSMAVLVAMILGGFSMSDLAKLAILMGATFAEMNTGGDVXXXXXXAAFKVRPALLVSFIF", 
      virus3 =&gt; "LGVLVILLMVQEGLKKRMTTKIIISTSMAVLVAMILGGFSMSDLAKLAILMGATFAEMNTGGZZZZZZLIAAFKVRPALLVSFIF"
  );
  
  # I define the start and end of the alignment positions I am splicing
  my <span class="math-container">$n = 1;
  my $</span>z = 5;
  
  # I turn the alignment into a string with separated by carriage returns 
  my @array = map {"<span class="math-container">$dna{$_}"} keys %dna;
  my $</span>string = join "\n", @array;
  
  # This is the awk loop, note in Perl the "``" means issue a shell command 
  
  my <span class="math-container">$splice = `echo "$</span>string" | awk 'BEGIN {FS=""} {x = <span class="math-container">$n; while (x &lt; $</span>z); print \$ x; ++x}' 2&gt;1`;
  print $splice;
  __DATA__
  </code></pre>
  
  <p><strong>Expected output</strong></p>
  
  <pre><code>LGVLV
  LGVLV
  LGVLV
  </code></pre>
  
  <p><strong>Observed output</strong>
  I get the original sequences returned</p>
  
  <p>I am familiar with Python/Perl solutions of line by line processing but awk is more efficient IMO. </p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-02-25T00:44:23.603Z
Comments.Text: Could you [edit] and show the output you're expecting? I speak both Perl and awk, but I don't get what output you want. Also, I really doubt that calling awk will be more efficient than doing it natively in perl. You don't seem to be using anything that awk is particularly good at.
Comments.UserDisplayName: terdon
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-02-25T00:49:36.620Z
Comments.Text: Amended, thanks. I will post the solution I got working, but this is because I don't understand awk
Comments.UserDisplayName: M__
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-02-25T00:54:53.143Z
Comments.Text: The main problem is that your `while` only encloses the first statement after it, and not the second. You need `{ }` for that. If you could explain what that awk is supposed to be doing (I'm pretty sure you've copied it wrong, is there really a space between `\$` and `x`?), I should be able to post an answer. Do you just want the first 5 characters of each sequence? Don't you want positions `$n` to `$z` (3-7)?
Comments.UserDisplayName: terdon
Tags: <perl><awk>
Title: awk for motif splicing from an amino acid alignment
ThreadText: # awk for motif splicing from an amino acid alignment
  
  (Asked by: username_0 on 2019-02-25)
  
  I'm looking to splice out specific amino acid alignment positions as a continuous motif from an alignment (~15 000 sequences). The code is written in Perl (I agree not many developers now) but splicing method is awk which a lot of you know. Please note the hash is "tied" (not shown), this just means the order of the sequences is preserved. I can get the awk code below to return single amino acid positions.
  
  I am struggling to get a loop working in awk.
  
  I agree the Perl wrapper makes it more complicated, but its part of a huge data pipeline (all written in Perl).
  
  ```perl
  #!/usr/local/ActivePerl-5.20/bin/env perl
  #use strict;
  use warnings;
  
  # I loaded the alignment into a hash, where key = sequence and value = protein
  my %dna = (
      virus1 => "LGVLVILLMVQEGLKKRMTTKIIISTSMAVLVAMILGGFSMSDLAKLAILMGATFAEMNTGGDVAHLALIAAFKVRPALLVSFIF", 
      virus2 => "LGVLVILLMVQEGLKKRMTTKIIISTSMAVLVAMILGGFSMSDLAKLAILMGATFAEMNTGGDVXXXXXXAAFKVRPALLVSFIF", 
      virus3 => "LGVLVILLMVQEGLKKRMTTKIIISTSMAVLVAMILGGFSMSDLAKLAILMGATFAEMNTGGZZZZZZLIAAFKVRPALLVSFIF"
  );
  
  # I define the start and end of the alignment positions I am splicing
  my $n = 1;
  my $z = 5;
  
  # I turn the alignment into a string with separated by carriage returns 
  my @array = map {"$dna{$_}"} keys %dna;
  my $string = join "\n", @array;
  
  # This is the awk loop, note in Perl the "``" means issue a shell command 
  
  my $splice = `echo "$string" | awk 'BEGIN {FS=""} {x = $n; while (x < $z); print \$ x; ++x}' 2>1`;
  print $splice;
  __DATA__
  
  ```
  
  **Expected output**
  
  ```
  LGVLV
  LGVLV
  LGVLV
  
  ```
  
  **Observed output**
  I get the original sequences returned
  
  I am familiar with Python/Perl solutions of line by line processing but awk is more efficient IMO.
  
  **Tags:** perl, awk
  
  > **Comments:**
  > **username_1**: Could you [edit] and show the output you're expecting? I speak both Perl and awk, but I don't get what output you want. Also, I really doubt that calling awk will be more efficient than doing it natively in perl. You don't seem to be using anything that awk is particularly good at.  
  > **username_0**: Amended, thanks. I will post the solution I got working, but this is because I don't understand awk  
  > **username_1**: The main problem is that your `while` only encloses the first statement after it, and not the second. You need `{ }` for that. If you could explain what that awk is supposed to be doing (I'm pretty sure you've copied it wrong, is there really a space between `\$` and `x`?), I should be able to post an answer. Do you just want the first 5 characters of each sequence? Don't you want positions `$n` to `$z` (3-7)?  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2019-02-25. Score: 1 | Accepted answer
  
  I *think* you are trying to use `awk` to get characters 3 through 7 (inclusive) of each input sequence. If so, a working version of your approach would be this:
  
  ```
  my $splice = `echo "$string" | awk 'BEGIN {FS=\"\"} {x = $n; while (x < $z){ printf \"%s\",\$x; ++x} print ""}' 2>1`;
  
  ```
  
  That, when added to your perl script, produces:
  
  ```
  $ foo.pl
  VLVIL
  VLVIL
  VLVIL
  
  ```
  
  That isn't the output you show, but the output you show is just the first 5 letters of each sequence, which doesn't seem to correspond to what your script is doing.
  
  Now, the reason your version didn't work, is that the `while` was like so:
  
  ```
  while (x < $z); print \$ x; ++x}
  
  ```
  
  Assuming the `\$ x` was a typo, the problem is that the `++x` isn't part of the while loop. So that should never even have exited. Seeing as we're in the same time zone, and given the time, I am guessing you're tired and that you miscopied something when trying to give us a reproducible example. In any case, to get the loop right, you need `{}`:
  
  ```
  while (x < $z){print \$x; ++x}
  
  ```
  
  However, if all you want is a substring, a loop is a very inefficient way of doing that. You can instead use `awk`'s `substr` command:
  
  ```
    substr(s, m[, n ])
               Return  the at most n-character substring of s that begins at position
               m, numbering from 1. If n is omitted, or if n specifies  more  charac‐
               ters than are left in the string, the length of the substring shall be
               limited by the length of the string s.
  
  ```
  
  So, in your script, that would be:
  
  ```
  my $splice = `echo "$string" | awk '{print substr(\$0,$n,5)}'
  
  ```
  
  ---
  
  Since you're already using a language that excels in string manipulation, I really don't understand why you would try to use `awk` instead. It will not be faster or more efficient than the same thing in perl:
  
  ```
  print substr($_,$n,5) . "\n" for @array;
  
  ```
  
  By calling `awk` you are also calling the shell which will be slower by definition, especially if you're `echo`ing thousands of lines. In fact, I tried this with 15000 sequences (just repetitions of your first one) and using either the `awk` loop I gave above, or the `awk`+`substr`, or the native perl `substr`:
  
  1. awk loop, average time taken to run the same thing 100 times: 0.0525
  2. awk substr, average time taken to run the same thing 100 times: 0.0522s
  3. perl substr, average time taken to run the same thing 100 times: 0.0464
  
  And you can make it even faster by skipping the array and iterating over the keys directly:
  
  ```
  print substr($dna{$_},$n,5) . "\n" for keys(%dna);
  
  ```
  
  That averaged at 0.0376s over 100 runs.
  
  > **Comments:**
  > **username_0**: Huge thanks @username_1. You are right I should have performed a runtime check :-( such as use Timer::Runtime. I worked on the implicit assumption awk was faster for this operation. David Cross "lectured" on runtime in detail in "Data Munging for Perl": There is no denying that 0.0525 vs. 0.0376s is 30% faster.  
  > **username_1**: @MichaelG. in my experience, awk is blindingly fast when your data are organized in fields because of its implicit splitting. If not, however, it doesn't give any real advantage over a language like Perl which is simply brilliant for text parsing.  
  
  ---
  
  ### Answer by username_2 on 2019-02-25. Score: 1
  
  If you have the sequences in a seperate file like this:
  
  ```
  $ cat input
  LGVLVILLMVQEGLKKRMTTKIIISTSMAVLVAMILGGFSMSDLAKLAILMGATFAEMNTGGDVAHLALIAAFKVRPALLVSFIF
  LGVLVILLMVQEGLKKRMTTKIIISTSMAVLVAMILGGFSMSDLAKLAILMGATFAEMNTGGDVXXXXXXAAFKVRPALLVSFIF
  LGVLVILLMVQEGLKKRMTTKIIISTSMAVLVAMILGGFSMSDLAKLAILMGATFAEMNTGGZZZZZZLIAAFKVRPALLVSFIF
  
  ```
  
  you can use a simple `cut` to fetch a substring:
  
  ```
  $ cut -c1-5 input
  LGVLV
  LGVLV
  LGVLV
  
  ```
  
  
  ---
  
  