Answers.Body: <p>The authors explain <em>their</em> use of the term in the paper:</p>
  <blockquote>
  <p>Without the bias correction we observe many degenerate runs, where fine-tuned models fail to outperform the random baseline</p>
  </blockquote>
  <p>Specifically, as it pertains to fine tuning, <em>degenerate</em> has this dictionary meaning in technical contexts: <em>Lacking some usual or expected property or quality</em>. In the paper's case the retrained classifier lacks the property of being able to classify inputs to any accuracy.</p>
  <p>That is backed up by their use of the term to apply to failed training runs.</p>
  <p>There <em>is</em> also a mathematics use of the term, which sometimes appears in computer science, and <a href="https://en.wikipedia.org/wiki/Degeneracy_(mathematics)" rel="nofollow noreferrer">is about object classes e.g. a triangle with one angle 0 degrees is degenerate - it is just a line, not meaningfully a triangle</a>. I don't think it applies here, although it could - the failed refined classifier has all the technical build qualities of a classifier, but does not classify.</p>
  <p>I do not think <em>degenerate runs</em> is a commonly used phrase in machine learning, nor does it have any additional technical meaning when applied to supervised learning, beyond what they explain in the paper.</p>
  <p>A more common term related to trained systems losing capabilities when trained further on new data is <em><a href="https://en.wikipedia.org/wiki/Catastrophic_interference" rel="nofollow noreferrer">catastrophic forgetting</a></em>. This may be subtly different to the effect that the authors are correcting for though, perhaps that is the reason for them using a different term.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-07T20:18:32.410Z
Answers.LastActivityDate: 2022-05-07T20:18:32.410Z
Answers.UserDisplayName: Neil Slater
Body: <p>I've recently come across <a href="https://arxiv.org/pdf/2006.05987.pdf" rel="nofollow noreferrer">a paper</a> that uses the term &quot;degenerate run&quot;, but I'm not sure if I understand what it means. The idea is that when they report the average performance of running fine-tuned models using multiple random seeds (e.g., a deep learning model where we need to initialize model parameters using multiple seeds to ensure results are robust,) they <em>exclude</em> the degenerate runs in some of their analyses.</p>
  <p>As this paper mentions, a degenerate run is &quot;where fine-tuned models fail to outperform the random baseline.&quot; But I wonder if this is a standard practice to eliminate such results when reporting the average performance? Or is the definition they give the correct meaning of a degenerate run?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-05-07T05:22:13.053Z
Comments.Text: Can you please cite the paper in the question?
Comments.UserDisplayName: hanugm
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-05-07T10:49:22.443Z
Comments.Text: Hello. Please, do as suggested in the previous comment. Include the link to and name of the paper you're mentioning.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-05-07T20:00:07.707Z
Comments.Text: @hanugm thanks for the suggestion. I just added the reference and more context.
Comments.UserDisplayName: Pedram
Tags: <machine-learning><terminology><papers><performance>
Title: What is a 'degenerate run' in evaluating model performance?
UserDisplayName: Pedram
ThreadText: # What is a 'degenerate run' in evaluating model performance?
  
  (Asked by: username_0 on 2022-05-07)
  
  I've recently come across [a paper](https://arxiv.org/pdf/2006.05987.pdf) that uses the term "degenerate run", but I'm not sure if I understand what it means. The idea is that when they report the average performance of running fine-tuned models using multiple random seeds (e.g., a deep learning model where we need to initialize model parameters using multiple seeds to ensure results are robust,) they *exclude* the degenerate runs in some of their analyses.
  
  As this paper mentions, a degenerate run is "where fine-tuned models fail to outperform the random baseline." But I wonder if this is a standard practice to eliminate such results when reporting the average performance? Or is the definition they give the correct meaning of a degenerate run?
  
  **Tags:** machine-learning, terminology, papers, performance
  
  > **Comments:**
  > **username_1**: Can you please cite the paper in the question?  
  > **username_2**: Hello. Please, do as suggested in the previous comment. Include the link to and name of the paper you're mentioning.  
  > **username_0**: @username_1 thanks for the suggestion. I just added the reference and more context.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2022-05-07. Score: 2
  
  The authors explain *their* use of the term in the paper:
  
  > Without the bias correction we observe many degenerate runs, where fine-tuned models fail to outperform the random baseline
  
  Specifically, as it pertains to fine tuning, *degenerate* has this dictionary meaning in technical contexts: *Lacking some usual or expected property or quality*. In the paper's case the retrained classifier lacks the property of being able to classify inputs to any accuracy.
  
  That is backed up by their use of the term to apply to failed training runs.
  
  There *is* also a mathematics use of the term, which sometimes appears in computer science, and [is about object classes e.g. a triangle with one angle 0 degrees is degenerate - it is just a line, not meaningfully a triangle](https://en.wikipedia.org/wiki/Degeneracy_(mathematics)). I don't think it applies here, although it could - the failed refined classifier has all the technical build qualities of a classifier, but does not classify.
  
  I do not think *degenerate runs* is a commonly used phrase in machine learning, nor does it have any additional technical meaning when applied to supervised learning, beyond what they explain in the paper.
  
  A more common term related to trained systems losing capabilities when trained further on new data is *[catastrophic forgetting](https://en.wikipedia.org/wiki/Catastrophic_interference)*. This may be subtly different to the effect that the authors are correcting for though, perhaps that is the reason for them using a different term.
  
  
  ---
  
  