Answers.Body: <p>Anyone can declare anything to be a &quot;gold-standard&quot;, the term is meaningless. All if actually means is, &quot;we think this is the test that everything else should be benchmarked against&quot;.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-07T12:14:03.937Z
Answers.LastActivityDate: 2021-01-07T12:14:03.937Z
Answers.UserDisplayName: Devon Ryan
Answers.Body: <p>Thanks for raising the issue, it is certainly topical. The question is whether the data set they unit tested is reliable.</p>
  <p>I think the issue they are trying get to is mosaic artefacts in the sequencing process and assembly thereon. This would be particularly true in a situation of <em>de novo</em> assembly which is what the authors are targeting. There are some slick methodologies for virus sequencing, which have been around for a while now, so in any case a 'gold-standard' would be outdated at present.</p>
  <p>I do have to agree with you that the language particularly for the journal certainly could have phrased better. The data set was 2014 and genome circularisation (its a virus small genomes) to remove the otherwise high SNP error rate would have been around in 2017 when this was published.</p>
  <p>I fully understand why the authors choose this language. A professional editor, not affiliated with the journal (without up to date technical insight), would likely recommend the same language.</p>
  <p>I guess the question would be whether verification is achieved on other virus data sets. There is no shortage of them at present and its certainly an interesting tool.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-28T21:27:19.557Z
Answers.Text: Well, I was not asking that because of any suspicion about their results. My software ([s-aligner](https://contignant.com/benchmarks)) gets better results for this benchmark set and I just wanted to know how relevant is that. It also outperforms other software in many other benchmarks.
Answers.UserDisplayName: juanjo75es
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-01T15:20:31.617Z
Answers.Text: Thanks @juanjo75es this is a good subject to explore in a question. The algorithms are pivotal to understanding viral diversity. I put this as a separate question and notify you. I'll explain what I understand and why within the question.
Answers.UserDisplayName: M__
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-09T01:04:50.793Z
Answers.Text: @juanjo75es I have raised the matter in the following question, https://bioinformatics.stackexchange.com/questions/18747/metagenomics-pipeline-recommendations
Answers.UserDisplayName: M__
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-28T14:42:33.053Z
Answers.LastActivityDate: 2022-02-28T15:09:13.157Z
Answers.LastEditDate: 2022-02-28T15:09:13.157Z
Answers.UserDisplayName: M__
Body: <p>This page is claimed to contain a gold standard benchmark for viral genome assembly.
  <a href="https://github.com/cbg-ethz/5-virus-mix" rel="nofollow noreferrer">https://github.com/cbg-ethz/5-virus-mix</a></p>
  <p>The claim is here:
  <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5411778/" rel="nofollow noreferrer">https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5411778/</a></p>
  <blockquote>
  <p>Baaijens <em>et al</em>. De novo assembly of viral quasispecies using overlap graphs
  Genome Research 2017 May; 27: 835–848.</p>
  </blockquote>
  <p>&quot;<em>This data set was recently presented as a gold standard benchmark (Di Giallonardo et al. 2014)</em>&quot;</p>
  <p>But at (Di Giallonardo et al. 2014) I can only find that mention:</p>
  <p>&quot;We previously estimated the frequencies of the virus strains in the 5-virus-mix by amplifying the protease gene using single-genome amplification (SGA), the current gold standard for studying diversity of virus populations&quot;.</p>
  <p>That doesn't seem to mean precisely that the dataset is a gold standard benchmark for genome assembly but, as I understand, that SGA is a gold standard for something else.</p>
  <p>My question is: is it? And who decided it is? Is still valid? Is there an official list of results somewhere for this benchmark set?</p>
  <p>I tried to contact the owner of the page but still no answer.</p>
  
Tags: <genome><assembly><covid-19><benchmarking><virus>
Title: Gold standard benchmark
ThreadText: # Gold standard benchmark
  
  (Asked by: username_0 on 2021-01-06)
  
  This page is claimed to contain a gold standard benchmark for viral genome assembly.
  <https://github.com/cbg-ethz/5-virus-mix>
  
  The claim is here:
  <https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5411778/>
  
  > Baaijens *et al*. De novo assembly of viral quasispecies using overlap graphs
  > Genome Research 2017 May; 27: 835–848.
  
  "*This data set was recently presented as a gold standard benchmark (Di Giallonardo et al. 2014)*"
  
  But at (Di Giallonardo et al. 2014) I can only find that mention:
  
  "We previously estimated the frequencies of the virus strains in the 5-virus-mix by amplifying the protease gene using single-genome amplification (SGA), the current gold standard for studying diversity of virus populations".
  
  That doesn't seem to mean precisely that the dataset is a gold standard benchmark for genome assembly but, as I understand, that SGA is a gold standard for something else.
  
  My question is: is it? And who decided it is? Is still valid? Is there an official list of results somewhere for this benchmark set?
  
  I tried to contact the owner of the page but still no answer.
  
  **Tags:** genome, assembly, covid-19, benchmarking, virus
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2021-01-07. Score: 1 | Accepted answer
  
  Anyone can declare anything to be a "gold-standard", the term is meaningless. All if actually means is, "we think this is the test that everything else should be benchmarked against".
  
  
  ---
  
  ### Answer by username_2 on 2022-02-28. Score: 1
  
  Thanks for raising the issue, it is certainly topical. The question is whether the data set they unit tested is reliable.
  
  I think the issue they are trying get to is mosaic artefacts in the sequencing process and assembly thereon. This would be particularly true in a situation of *de novo* assembly which is what the authors are targeting. There are some slick methodologies for virus sequencing, which have been around for a while now, so in any case a 'gold-standard' would be outdated at present.
  
  I do have to agree with you that the language particularly for the journal certainly could have phrased better. The data set was 2014 and genome circularisation (its a virus small genomes) to remove the otherwise high SNP error rate would have been around in 2017 when this was published.
  
  I fully understand why the authors choose this language. A professional editor, not affiliated with the journal (without up to date technical insight), would likely recommend the same language.
  
  I guess the question would be whether verification is achieved on other virus data sets. There is no shortage of them at present and its certainly an interesting tool.
  
  > **Comments:**
  > **username_0**: Well, I was not asking that because of any suspicion about their results. My software ([s-aligner](https://contignant.com/benchmarks)) gets better results for this benchmark set and I just wanted to know how relevant is that. It also outperforms other software in many other benchmarks.  
  > **username_2**: Thanks @username_0 this is a good subject to explore in a question. The algorithms are pivotal to understanding viral diversity. I put this as a separate question and notify you. I'll explain what I understand and why within the question.  
  > **username_2**: @username_0 I have raised the matter in the following question, https://bioinformatics.stackexchange.com/questions/18747/metagenomics-pipeline-recommendations  
  
  ---
  
  