Answers.Body: <p><code>--thread 4</code> is essential, <code>--maxiterate</code> just try and get it working, not getting it to work optimally. Just make sure there are no other processes running on the machine when --thread 4 is operating, e.g. you are using another machine as your desktop</p>
  <p>The gist is as you spread the calculation across your processors (4) and simplify the calculation (lowering the iterations), you are getting further into the calculation, for example now you are getting into gap penalties.</p>
  <p>If you are agreed then I would try performing the alignment using just one chromosome of the human genome against the chimp genome and I suspect that would work. Then take it from there.</p>
  <p>The objective is to get the calculation working, for example I wouldn't trust a result with max. iterations of 2, but once you have it working you can then start refining the parameters towards optimal results. The i3 processor isn't perfect in context.</p>
  <p>Using <code>--thread 4</code> is super important and you want to increase maxiterate towards the default if the machine supports it.</p>
  <p>I suspect the solution is to perform the calculation on an 8+-core machine and set <code>--thread 8</code> (or higher depending on the number of cores on the machine), at least it will get further into the calculation. With 4-cores and an i3 (diagnostics of RAM usage would be nice) in any case it just doesn't seem to like it. The alternative is to perform this across the genome via sampling the chromosomes on your current i3.</p>
  <hr />
  <p>Just to explain why this works. When you have a RAM intensive job and you spread it across cores the RAM overhead massively reduces for each core that is added. Don't ask me how this works in the hardware mecahanics of RAM distribution per core. The reason I know this is because of emperically parallelising Bayesian calculation using each chain on a different core.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-20T14:48:03.703Z
Answers.Text: I am actually comparing all chromosomes of human vs all chromosomes of chimp. I doesn't clearly understand what do you mean. You chould try it yourself I can wait
Answers.UserDisplayName: Alexey
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-20T14:56:03.770Z
Answers.Text: Hi @Alexey you have to get it working first. Only when it is working can you move towards the calculation you want. This is not the finished product just a means of verifying it will work. One issue is the i3 processor in the machine. I wouldn't trust a result with max iterations at 2, but once it works you would increase that parameter.
Answers.UserDisplayName: M__
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-20T15:04:36.500Z
Answers.Text: Basically to cut to the chase, I think your machine is underpowered for the calculation you are performing.
Answers.UserDisplayName: M__
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-20T14:43:43.903Z
Answers.LastActivityDate: 2020-06-20T18:35:44.533Z
Answers.LastEditDate: 2020-06-20T18:35:44.533Z
Answers.UserDisplayName: M__
Body: <p>I have 2 .fa sequenes like:</p>
  <pre><code>&gt;some description of first...
  ACTGACTACGTACGACTACGATCTGACTACACGTAGATAGACTAGTCACTACG
  ACTGACTACGTACGACTGACTACATGACTGAC...
  &gt; some description of second...
  ATCGATAGCTACGATCGATAGTAGATAG....
  </code></pre>
  <p>This file is 461mb (1 chromosome of homo sapiens &amp; 1 chromosome of pan troglodytes) and I want to align it. I am using mafft.</p>
  <pre><code>mafft --globalpair --maxiterate 1000 Homo_sapiens_1_Pan_troglodytes_1.fa &gt; ./mafft_results/Homo_sapiens_1_Pan_troglodytes_1_output.fa
  </code></pre>
  <p>And it returns me</p>
  <pre><code>generating a scoring matrix for nucleotide (dist=200) ... done
  All-to-all alignment.
  /usr/bin/mafft: line 2719:  6030 Killed                  &quot;<span class="math-container">$prefix/tbfast" _ -u $</span>unalignlevel <span class="math-container">$localparam -C $</span>numthreads <span class="math-container">$seqtype $</span>model -g <span class="math-container">$pgexp -f $</span>pggop -Q <span class="math-container">$spfactor -h $</span>pgaof -A <span class="math-container">$usenaivepairscore $</span>focusarg _ -+ <span class="math-container">$iterate -W $</span>minimumweight -V &quot;-&quot;<span class="math-container">$gopdist -s $</span>unalignlevel <span class="math-container">$legacygapopt $</span>mergearg <span class="math-container">$outnum $</span>addarg <span class="math-container">$add2ndhalfarg -C $</span>numthreadstb <span class="math-container">$rnaopt $</span>weightopt <span class="math-container">$treeinopt $</span>treeoutopt <span class="math-container">$distoutopt $</span>seqtype <span class="math-container">$model -f "-"$</span>gop -Q <span class="math-container">$spfactor -h $</span>aof <span class="math-container">$param_fft $</span>localparam <span class="math-container">$algopt $</span>treealg <span class="math-container">$scoreoutarg $</span>focusarg &lt; infile &gt; /dev/null 2&gt;&gt; &quot;$progressfile&quot;
  </code></pre>
  <p>Why? How could I align it for further comparing?</p>
  <p><strong>EDIT 1</strong></p>
  <p>MAFFT working cool, when I tried to use it on test sequence file</p>
  <pre><code>&gt;test1
  ATCGATCGATGACTACGATCGACTACGTACG
  ACTAGCTAGCTACGCATAGTAGATCGA
  ATCGACTAGCTCAGATCGACTAGATCGATCGT
  &gt;test2
  TTATCCACTGATCGACTCAGATCGACTCAGAC
  ATCGATCAGCTACGATGACTACGTAC
  ACTGACTAGCTAGACTGACTAGCTACGTCA
  </code></pre>
  <p>and after MAFFT <code>mafft test.fa &gt; output.fa</code> it returned</p>
  <blockquote>
  <p>test1
  ------atcgatcgat-------------gactacgatcgactacgtacgactagctagc
  tacgcatagtagatcgaatcgactagctcagatcgactagatcgatcgt
  test2
  ttatccactgatcgactcagatcgactcagacatcgatcagctac---------gatgac
  tacgtac----------actgactagct-agactgactagctacgtca-</p>
  </blockquote>
  <p>So it works.</p>
  <p>I can use all 4 at the example. Now about my large file:</p>
  <pre><code>alex@alex:~/Desktop/test/BOTH/homo_sapiens_pan_troglodytes$ mafft --thread 4 --maxiterate 2 Homo_sapiens_1_Pan_troglodytes_1.fa &gt; ./mafft_results/Homo_sapiens_1_Pan_troglodytes_1_output.fa
  nthread = 4
  nthreadpair = 4
  nthreadtb = 4
  ppenalty_ex = 0
  reallocating...
  done.
  reallocating...
  done.
  reallocating...
      done.
  reallocating...
  done.
  reallocating...
  ...
  stacksize: 8192 kb
  reallocating...
  done.
  reallocating...
  done.
  reallocating...
  done.
  ...
  generating a scoring matrix for nucleotide (dist=200) ... done
  Gap Penalty = -1.53, +0.00, +0.00
  
  
  
  Making a distance matrix ..
      1 / 2 (thread    1)
  done.
  
  Constructing a UPGMA tree (efffree=1) ... 
      0 / 2
  done.
  
  Progressive alignment 1/1... 
  
  len1=230480994, len2=223796828, Switching to the memsave mode
  STEP     1 / 1 (thread    0)m/usr/bin/mafft: line 2719:  3192 Killed                  &quot;<span class="math-container">$prefix/disttbfast" -q $</span>npickup -E <span class="math-container">$cycledisttbfast -V "-"$</span>gopdist -s <span class="math-container">$unalignlevel $</span>legacygapopt <span class="math-container">$mergearg -W $</span>tuplesize <span class="math-container">$termgapopt $</span>outnum <span class="math-container">$addarg $</span>add2ndhalfarg -C <span class="math-container">$numthreads-$</span>numthreadstb <span class="math-container">$memopt $</span>weightopt <span class="math-container">$treeinopt $</span>treeoutopt <span class="math-container">$distoutopt $</span>seqtype <span class="math-container">$model -g $</span>gexp -f &quot;-&quot;<span class="math-container">$gop -Q $</span>spfactor -h <span class="math-container">$aof $</span>param_fft <span class="math-container">$algopt $</span>treealg <span class="math-container">$scoreoutarg $</span>anchoropt &lt; infile &gt; pre 2&gt;&gt; &quot;$progressfile&quot;
  </code></pre>
  <p>Doesnt matter I do not enter <code>--thread 4</code> or not, <code>--maxiterate 2</code> or <code>--maxiterate 1000</code>, it still shows me same error. It takes around 2 minute. Ram usage when running at the first minute is 6656mb, after 1 min 11028mb and then it out of ram and error..</p>
  <p><em>PS</em></p>
  <p>discussion moved <a href="https://stackoverflow.com/questions/62488623/out-of-ram-while-work-with-very-large-matrix">here</a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-20T12:46:22.600Z
Comments.Text: What machine are you running this on, particularly RAM capacity? Just on a remote off chance are there any "core" dumps, e.g. weirdly large files that suddenly 'appear', especially if they have the name 'core' :-) ?
Comments.UserDisplayName: M__
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-20T13:00:04.487Z
Comments.Text: I am running this on  Ubuntu 20.04, CPU(s): 4, RAM 12 GB, Intel(R) Core(TM) i3-3220.. Do not think that this file could be so large for 12gb ram..
Comments.UserDisplayName: Alexey
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-20T13:11:35.730Z
Comments.Text: I'd do a full diagnostic and post back, because the first stage is obviously working. E.g. have you successfully got MAFFT working on any other data set?  Are all processors being used? Whats the RAM usage when running? Do any other long processes/calculations suddenly quit (e.g. hardware bug)? How long before the error pops up.  What happens when you greatly lower the maxiterations?
Comments.UserDisplayName: M__
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-20T13:38:20.507Z
Comments.Text: @Michael please see my edit 1. But I still want to compare them
Comments.UserDisplayName: Alexey
Tags: <phylogenetics><phylogeny><sequence-alignment>
Title: Can't align 2 sequences, (MAFFT killed)
ThreadText: # Can't align 2 sequences, (MAFFT killed)
  
  (Asked by: username_0 on 2020-06-20)
  
  I have 2 .fa sequenes like:
  
  ```
  >some description of first...
  ACTGACTACGTACGACTACGATCTGACTACACGTAGATAGACTAGTCACTACG
  ACTGACTACGTACGACTGACTACATGACTGAC...
  > some description of second...
  ATCGATAGCTACGATCGATAGTAGATAG....
  
  ```
  
  This file is 461mb (1 chromosome of homo sapiens & 1 chromosome of pan troglodytes) and I want to align it. I am using mafft.
  
  ```
  mafft --globalpair --maxiterate 1000 Homo_sapiens_1_Pan_troglodytes_1.fa > ./mafft_results/Homo_sapiens_1_Pan_troglodytes_1_output.fa
  
  ```
  
  And it returns me
  
  ```
  generating a scoring matrix for nucleotide (dist=200) ... done
  All-to-all alignment.
  /usr/bin/mafft: line 2719:  6030 Killed                  "$prefix/tbfast" _ -u $unalignlevel $localparam -C $numthreads $seqtype $model -g $pgexp -f $pggop -Q $spfactor -h $pgaof -A $usenaivepairscore $focusarg _ -+ $iterate -W $minimumweight -V "-"$gopdist -s $unalignlevel $legacygapopt $mergearg $outnum $addarg $add2ndhalfarg -C $numthreadstb $rnaopt $weightopt $treeinopt $treeoutopt $distoutopt $seqtype $model -f "-"$gop -Q $spfactor -h $aof $param_fft $localparam $algopt $treealg $scoreoutarg $focusarg < infile > /dev/null 2>> "$progressfile"
  
  ```
  
  Why? How could I align it for further comparing?
  
  **EDIT 1**
  
  MAFFT working cool, when I tried to use it on test sequence file
  
  ```
  >test1
  ATCGATCGATGACTACGATCGACTACGTACG
  ACTAGCTAGCTACGCATAGTAGATCGA
  ATCGACTAGCTCAGATCGACTAGATCGATCGT
  >test2
  TTATCCACTGATCGACTCAGATCGACTCAGAC
  ATCGATCAGCTACGATGACTACGTAC
  ACTGACTAGCTAGACTGACTAGCTACGTCA
  
  ```
  
  and after MAFFT `mafft test.fa > output.fa` it returned
  
  > test1
  > ------atcgatcgat-------------gactacgatcgactacgtacgactagctagc
  > tacgcatagtagatcgaatcgactagctcagatcgactagatcgatcgt
  > test2
  > ttatccactgatcgactcagatcgactcagacatcgatcagctac---------gatgac
  > tacgtac----------actgactagct-agactgactagctacgtca-
  
  So it works.
  
  I can use all 4 at the example. Now about my large file:
  
  ```
  alex@alex:~/Desktop/test/BOTH/homo_sapiens_pan_troglodytes$ mafft --thread 4 --maxiterate 2 Homo_sapiens_1_Pan_troglodytes_1.fa > ./mafft_results/Homo_sapiens_1_Pan_troglodytes_1_output.fa
  nthread = 4
  nthreadpair = 4
  nthreadtb = 4
  ppenalty_ex = 0
  reallocating...
  done.
  reallocating...
  done.
  reallocating...
      done.
  reallocating...
  done.
  reallocating...
  ...
  stacksize: 8192 kb
  reallocating...
  done.
  reallocating...
  done.
  reallocating...
  done.
  ...
  generating a scoring matrix for nucleotide (dist=200) ... done
  Gap Penalty = -1.53, +0.00, +0.00
  
  Making a distance matrix ..
      1 / 2 (thread    1)
  done.
  
  Constructing a UPGMA tree (efffree=1) ... 
      0 / 2
  done.
  
  Progressive alignment 1/1... 
  
  len1=230480994, len2=223796828, Switching to the memsave mode
  STEP     1 / 1 (thread    0)m/usr/bin/mafft: line 2719:  3192 Killed                  "$prefix/disttbfast" -q $npickup -E $cycledisttbfast -V "-"$gopdist -s $unalignlevel $legacygapopt $mergearg -W $tuplesize $termgapopt $outnum $addarg $add2ndhalfarg -C $numthreads-$numthreadstb $memopt $weightopt $treeinopt $treeoutopt $distoutopt $seqtype $model -g $gexp -f "-"$gop -Q $spfactor -h $aof $param_fft $algopt $treealg $scoreoutarg $anchoropt < infile > pre 2>> "$progressfile"
  
  ```
  
  Doesnt matter I do not enter `--thread 4` or not, `--maxiterate 2` or `--maxiterate 1000`, it still shows me same error. It takes around 2 minute. Ram usage when running at the first minute is 6656mb, after 1 min 11028mb and then it out of ram and error..
  
  *PS*
  
  discussion moved [here](https://stackoverflow.com/questions/62488623/out-of-ram-while-work-with-very-large-matrix)
  
  **Tags:** phylogenetics, phylogeny, sequence-alignment
  
  > **Comments:**
  > **username_1**: What machine are you running this on, particularly RAM capacity? Just on a remote off chance are there any "core" dumps, e.g. weirdly large files that suddenly 'appear', especially if they have the name 'core' :-) ?  
  > **username_0**: I am running this on  Ubuntu 20.04, CPU(s): 4, RAM 12 GB, Intel(R) Core(TM) i3-3220.. Do not think that this file could be so large for 12gb ram..  
  > **username_1**: I'd do a full diagnostic and post back, because the first stage is obviously working. E.g. have you successfully got MAFFT working on any other data set?  Are all processors being used? Whats the RAM usage when running? Do any other long processes/calculations suddenly quit (e.g. hardware bug)? How long before the error pops up.  What happens when you greatly lower the maxiterations?  
  > **username_0**: @Michael please see my edit 1. But I still want to compare them  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-06-20. Score: 1
  
  `--thread 4` is essential, `--maxiterate` just try and get it working, not getting it to work optimally. Just make sure there are no other processes running on the machine when --thread 4 is operating, e.g. you are using another machine as your desktop
  
  The gist is as you spread the calculation across your processors (4) and simplify the calculation (lowering the iterations), you are getting further into the calculation, for example now you are getting into gap penalties.
  
  If you are agreed then I would try performing the alignment using just one chromosome of the human genome against the chimp genome and I suspect that would work. Then take it from there.
  
  The objective is to get the calculation working, for example I wouldn't trust a result with max. iterations of 2, but once you have it working you can then start refining the parameters towards optimal results. The i3 processor isn't perfect in context.
  
  Using `--thread 4` is super important and you want to increase maxiterate towards the default if the machine supports it.
  
  I suspect the solution is to perform the calculation on an 8+-core machine and set `--thread 8` (or higher depending on the number of cores on the machine), at least it will get further into the calculation. With 4-cores and an i3 (diagnostics of RAM usage would be nice) in any case it just doesn't seem to like it. The alternative is to perform this across the genome via sampling the chromosomes on your current i3.
  
  ---
  
  Just to explain why this works. When you have a RAM intensive job and you spread it across cores the RAM overhead massively reduces for each core that is added. Don't ask me how this works in the hardware mecahanics of RAM distribution per core. The reason I know this is because of emperically parallelising Bayesian calculation using each chain on a different core.
  
  > **Comments:**
  > **username_0**: I am actually comparing all chromosomes of human vs all chromosomes of chimp. I doesn't clearly understand what do you mean. You chould try it yourself I can wait  
  > **username_1**: Hi @username_0 you have to get it working first. Only when it is working can you move towards the calculation you want. This is not the finished product just a means of verifying it will work. One issue is the i3 processor in the machine. I wouldn't trust a result with max iterations at 2, but once it works you would increase that parameter.  
  > **username_1**: Basically to cut to the chase, I think your machine is underpowered for the calculation you are performing.  
  
  ---
  
  