Answers.Body: <p>The <strong>mmseqs2</strong> code should be specifically compiled and optimised for your system. <strong>mmseqs2</strong> optimisation is automated and might be available via Spack* (I'm not sure its actually done for mmseqs2), it is definitely available for <strong>EasyBuild</strong>. I wouldn't advise attempting a manual optimisation unless you've prior experience.</p>
  <p>How much time this will save is more difficult to assess but 50% faster is a conservative assessment. This approach is good practice for a job like this size and its good for the associated report, its standard HPC practice. If you're doing cloud computation (i.e. $$$), HPC optimisation is essential. AWS make serious money from suboptimal HPC practice (in fairness to them they also do a lot of development work to minimise costs).</p>
  <p>Finally, if you're an expert in optimisation you can save <strong>serious</strong> computational time, but this is a major investment of your personal development time and technique. The rationale of how and why is complicated.</p>
  <p>*, its very likely available on AWS.</p>
  <hr />
  <p>You might simply consider its been running for a few days already so HPC technique isn't needed for this specific job. However, you'll know for the future.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-21T13:16:28.630Z
Answers.Text: Is the tool multithreaded? The OP mentioned "CPU is used at the maximum" in `htop` which makes me think that it might be 100% of a single core.
Answers.UserDisplayName: terdon
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-21T13:19:54.953Z
Answers.Text: @terdon. Yes it is. Whether it's default - dunno - I was thinking that too. Which is one reason why I requested the `htop`, but OPs response strongly suggests the parallelisation is default. It's easy to code parallelisation as default (at least in Python), normally its *Total CPU - 1*. What is a bit unusual I'll admit ... one CPU should be free (standard coding practice) but the OP didn't report that. However, we don't know how the original coder did this. To be honest **mmseqs** coder is hot, so they'd know the convention. If **mmseqs2** is intended for HPC *Total CPU - 1* is dropped
Answers.UserDisplayName: M__
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-21T12:51:00.787Z
Answers.LastActivityDate: 2023-11-21T13:17:11.010Z
Answers.LastEditDate: 2023-11-21T13:17:11.010Z
Answers.UserDisplayName: M__
Answers.Body: <p>When I want to estimate the time taken to do something big, I subset my input data to something that takes a few minutes or less, and calculate forward from there. If there's alignment involved, it's usually a good idea to work it out for multiple dataset sizes, as there could be a linear relationship between input size and time taken, square relationship, or worse.</p>
  <p>Have you looked at how long it takes for, say, 1000 sequences, 2000 sequences, and 5000 sequences?</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-21T08:05:33.697Z
Answers.LastActivityDate: 2023-11-21T08:05:33.697Z
Answers.UserDisplayName: gringer
Body: <p>I have the Uniref90 dataset containing around 76M sequences. I have made modifications to all sequences based on a reduced amino acid alphabet.</p>
  <p>I am trying to cluster it again at 90% identity. I am using  <code>mmseq cluster</code> on a 64 core, 500GB RAM machine. It has been more than 12h now.</p>
  <pre><code>Starting prefiltering scores calculation (step 1 of 1)
  Query db start 1 to 74390789
  Target db start 1 to 74390789
  [=
  </code></pre>
  <p>How long does it normally take to cluster such a database? I know I can use <code>linclust</code> instead to cluster in linear time, but it might not be what I need in this case.</p>
  <p>I am using <code>mmseq cluster</code> as follows:</p>
  <pre><code>mmseqs createdb &quot;uniref90-$1.fasta&quot; &quot;uniref90-DB-$1&quot;
  mmseqs cluster &quot;uniref90-DB-$1&quot;  &quot;uniref90-DB-$1_clu&quot; tmp/ --cov-mode 2 -c 0.8 --min-seq-id 0.9
  </code></pre>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-11-21T00:25:12.180Z
Comments.Text: Thanks. Hmmm ... when you use `htop` what do you see when its running?
Comments.UserDisplayName: M__
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-11-21T11:57:19.867Z
Comments.Text: CPU is used at the maximum and 20% of RAM. Anyway, I have looked into the mmseqs2 in more detail and found that it should take at least a few days.
Comments.UserDisplayName: GIONII
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-11-21T13:07:40.720Z
Comments.Text: Thanks and thank you for your question. Thats very clear and answered below.
Comments.UserDisplayName: M__
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-11-21T13:21:17.980Z
Comments.Text: What does "CPU is used at the maximum" mean? Do you see 100% in the CPU% column? Or do you see 6400%?
Comments.UserDisplayName: terdon
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-11-22T13:17:18.363Z
Comments.Text: 6400 %CPU, 12.8 %MEM
Comments.UserDisplayName: GIONII
Tags: <proteins><clustering><uniprot>
Title: How long should it take to cluster Uniref?
ThreadText: # How long should it take to cluster Uniref?
  
  (Asked by: username_0 on 2023-11-20)
  
  I have the Uniref90 dataset containing around 76M sequences. I have made modifications to all sequences based on a reduced amino acid alphabet.
  
  I am trying to cluster it again at 90% identity. I am using `mmseq cluster` on a 64 core, 500GB RAM machine. It has been more than 12h now.
  
  ```
  Starting prefiltering scores calculation (step 1 of 1)
  Query db start 1 to 74390789
  Target db start 1 to 74390789
  [=
  
  ```
  
  How long does it normally take to cluster such a database? I know I can use `linclust` instead to cluster in linear time, but it might not be what I need in this case.
  
  I am using `mmseq cluster` as follows:
  
  ```
  mmseqs createdb "uniref90-$1.fasta" "uniref90-DB-$1"
  mmseqs cluster "uniref90-DB-$1"  "uniref90-DB-$1_clu" tmp/ --cov-mode 2 -c 0.8 --min-seq-id 0.9
  
  ```
  
  **Tags:** proteins, clustering, uniprot
  
  > **Comments:**
  > **username_1**: Thanks. Hmmm ... when you use `htop` what do you see when its running?  
  > **username_0**: CPU is used at the maximum and 20% of RAM. Anyway, I have looked into the mmseqs2 in more detail and found that it should take at least a few days.  
  > **username_1**: Thanks and thank you for your question. Thats very clear and answered below.  
  > **username_2**: What does "CPU is used at the maximum" mean? Do you see 100% in the CPU% column? Or do you see 6400%?  
  > **username_0**: 6400 %CPU, 12.8 %MEM  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2023-11-21. Score: 0
  
  The **mmseqs2** code should be specifically compiled and optimised for your system. **mmseqs2** optimisation is automated and might be available via Spack\* (I'm not sure its actually done for mmseqs2), it is definitely available for **EasyBuild**. I wouldn't advise attempting a manual optimisation unless you've prior experience.
  
  How much time this will save is more difficult to assess but 50% faster is a conservative assessment. This approach is good practice for a job like this size and its good for the associated report, its standard HPC practice. If you're doing cloud computation (i.e. $$$), HPC optimisation is essential. AWS make serious money from suboptimal HPC practice (in fairness to them they also do a lot of development work to minimise costs).
  
  Finally, if you're an expert in optimisation you can save **serious** computational time, but this is a major investment of your personal development time and technique. The rationale of how and why is complicated.
  
  \*, its very likely available on AWS.
  
  ---
  
  You might simply consider its been running for a few days already so HPC technique isn't needed for this specific job. However, you'll know for the future.
  
  > **Comments:**
  > **username_2**: Is the tool multithreaded? The OP mentioned "CPU is used at the maximum" in `htop` which makes me think that it might be 100% of a single core.  
  > **username_1**: @username_2. Yes it is. Whether it's default - dunno - I was thinking that too. Which is one reason why I requested the `htop`, but OPs response strongly suggests the parallelisation is default. It's easy to code parallelisation as default (at least in Python), normally its *Total CPU - 1*. What is a bit unusual I'll admit ... one CPU should be free (standard coding practice) but the OP didn't report that. However, we don't know how the original coder did this. To be honest **mmseqs** coder is hot, so they'd know the convention. If **mmseqs2** is intended for HPC *Total CPU - 1* is dropped  
  
  ---
  
  ### Answer by username_3 on 2023-11-21. Score: 0
  
  When I want to estimate the time taken to do something big, I subset my input data to something that takes a few minutes or less, and calculate forward from there. If there's alignment involved, it's usually a good idea to work it out for multiple dataset sizes, as there could be a linear relationship between input size and time taken, square relationship, or worse.
  
  Have you looked at how long it takes for, say, 1000 sequences, 2000 sequences, and 5000 sequences?
  
  
  ---
  
  