Answers.Body: <p>In short: you should provide the optimal (best) value, but this is not the whole story.</p>
  <p>As you said:</p>
  <blockquote>
  <p>I think sometimes the optimal value is a result of randomness.</p>
  </blockquote>
  <p>This is perfectly true. Therefore, if you want to provide high-quality research results, you <em>should NOT</em> consider the single best value due to randomness. You may have been just luck, and got a very high value which, in reality, can be a very rare training run.</p>
  <p>What you should do is to <em>train the model multiple times</em> according to your computational budget: when the dependence on random numbers is not too strong, even <span class="math-container">$3$</span>, <span class="math-container">$5$</span>, or <span class="math-container">$10$</span> training runs (with different random seeds) are enough for statistically relevant results. But when randomness is an issue, like in reinforcement learning, way more runs (even 100+) may be necessary.</p>
  <p>In this way you can aggregate the result by <em>mean</em> (i.e, average expected performance of your model class) and <em>stddedv</em> (expected deviation from optimal model.) Doing so, you can obtain much more reliable results.</p>
  <ul>
  <li>Another option is to plot the results with associated <em>error bars</em>, such that the wider the bar is the more the variance is.</li>
  <li>A further one, is to get a <em>p-value</em> (by comparing the distribution of results) which is interpreted as the probability that method (or algorithm) A beats method B.</li>
  </ul>
  <p>(b.t.w. I also assume your training and evaluation setups are mostly, if not entirely, <em>reproducible</em> - i.e., deterministic given a fixed random seed.)</p>
  <p>As a side fact: is also important to train until your model converges, so until the learning curve stabilizes around a certain value (which is the best one so far, or very close.) This is important to check that both your model and training procedure is healty.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-23T17:38:33.610Z
Answers.LastActivityDate: 2023-12-24T12:05:03.890Z
Answers.LastEditDate: 2023-12-24T12:05:03.890Z
Answers.UserDisplayName: Luca Anzalone
Body: <p>When evaluating the performance of a deep learning model (for the purpose of publishing a research work), should we choose the optimal value from the metric curve (such as the accuracy curve) or the convergence value? I think sometimes the optimal value is a result of randomness. So I'm not sure if it can reflect the actual performance of a model. For example, in the accuracy-epoch curve below, should I choose the optimal value (highlighted by the red circle) as the model performance?
  <a href="https://i.sstatic.net/V6P1U.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/V6P1U.png" alt="enter image description here" /></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-01-02T13:12:45.700Z
Comments.Text: Just to clarify, are you asking if you should use the model that corresponds to the optimal metric (e.g. accuracy in this case), right?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-01-03T15:57:01.857Z
Comments.Text: @nbro Yes, exactly.
Comments.UserDisplayName: Liuji
Tags: <machine-learning><deep-learning><metric><testing>
Title: How to choose the metric value when evaluating the performance of a deep learning model?
UserDisplayName: Liuji
ThreadText: # How to choose the metric value when evaluating the performance of a deep learning model?
  
  (Asked by: username_0 on 2023-12-23)
  
  When evaluating the performance of a deep learning model (for the purpose of publishing a research work), should we choose the optimal value from the metric curve (such as the accuracy curve) or the convergence value? I think sometimes the optimal value is a result of randomness. So I'm not sure if it can reflect the actual performance of a model. For example, in the accuracy-epoch curve below, should I choose the optimal value (highlighted by the red circle) as the model performance?
  [![enter image description here](https://i.sstatic.net/V6P1U.png)](https://i.sstatic.net/V6P1U.png)
  
  **Tags:** machine-learning, deep-learning, metric, testing
  
  > **Comments:**
  > **username_1**: Just to clarify, are you asking if you should use the model that corresponds to the optimal metric (e.g. accuracy in this case), right?  
  > **username_0**: @username_1 Yes, exactly.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-12-23. Score: 2
  
  In short: you should provide the optimal (best) value, but this is not the whole story.
  
  As you said:
  
  > I think sometimes the optimal value is a result of randomness.
  
  This is perfectly true. Therefore, if you want to provide high-quality research results, you *should NOT* consider the single best value due to randomness. You may have been just luck, and got a very high value which, in reality, can be a very rare training run.
  
  What you should do is to *train the model multiple times* according to your computational budget: when the dependence on random numbers is not too strong, even $3$, $5$, or $10$ training runs (with different random seeds) are enough for statistically relevant results. But when randomness is an issue, like in reinforcement learning, way more runs (even 100+) may be necessary.
  
  In this way you can aggregate the result by *mean* (i.e, average expected performance of your model class) and *stddedv* (expected deviation from optimal model.) Doing so, you can obtain much more reliable results.
  
  * Another option is to plot the results with associated *error bars*, such that the wider the bar is the more the variance is.
  * A further one, is to get a *p-value* (by comparing the distribution of results) which is interpreted as the probability that method (or algorithm) A beats method B.
  
  (b.t.w. I also assume your training and evaluation setups are mostly, if not entirely, *reproducible* - i.e., deterministic given a fixed random seed.)
  
  As a side fact: is also important to train until your model converges, so until the learning curve stabilizes around a certain value (which is the best one so far, or very close.) This is important to check that both your model and training procedure is healty.
  
  
  ---
  
  