<p>In short: you should provide the optimal (best) value, but this is not the whole story.</p>
<p>As you said:</p>
<blockquote>
<p>I think sometimes the optimal value is a result of randomness.</p>
</blockquote>
<p>This is perfectly true. Therefore, if you want to provide high-quality research results, you <em>should NOT</em> consider the single best value due to randomness. You may have been just luck, and got a very high value which, in reality, can be a very rare training run.</p>
<p>What you should do is to <em>train the model multiple times</em> according to your computational budget: when the dependence on random numbers is not too strong, even <span class="math-container">$3$</span>, <span class="math-container">$5$</span>, or <span class="math-container">$10$</span> training runs (with different random seeds) are enough for statistically relevant results. But when randomness is an issue, like in reinforcement learning, way more runs (even 100+) may be necessary.</p>
<p>In this way you can aggregate the result by <em>mean</em> (i.e, average expected performance of your model class) and <em>stddedv</em> (expected deviation from optimal model.) Doing so, you can obtain much more reliable results.</p>
<ul>
<li>Another option is to plot the results with associated <em>error bars</em>, such that the wider the bar is the more the variance is.</li>
<li>A further one, is to get a <em>p-value</em> (by comparing the distribution of results) which is interpreted as the probability that method (or algorithm) A beats method B.</li>
</ul>
<p>(b.t.w. I also assume your training and evaluation setups are mostly, if not entirely, <em>reproducible</em> - i.e., deterministic given a fixed random seed.)</p>
<p>As a side fact: is also important to train until your model converges, so until the learning curve stabilizes around a certain value (which is the best one so far, or very close.) This is important to check that both your model and training procedure is healty.</p>

CC BY-SA 4.0
2023-12-23T17:38:33.610Z
2023-12-24T12:05:03.890Z
2023-12-24T12:05:03.890Z
Luca Anzalone
<p>When evaluating the performance of a deep learning model (for the purpose of publishing a research work), should we choose the optimal value from the metric curve (such as the accuracy curve) or the convergence value? I think sometimes the optimal value is a result of randomness. So I'm not sure if it can reflect the actual performance of a model. For example, in the accuracy-epoch curve below, should I choose the optimal value (highlighted by the red circle) as the model performance?
<a href="https://i.sstatic.net/V6P1U.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/V6P1U.png" alt="enter image description here" /></a></p>

CC BY-SA 4.0
2024-01-02T13:12:45.700Z
Just to clarify, are you asking if you should use the model that corresponds to the optimal metric (e.g. accuracy in this case), right?
nbro
CC BY-SA 4.0
2024-01-03T15:57:01.857Z
@nbro Yes, exactly.
Liuji
<machine-learning><deep-learning><metric><testing>
How to choose the metric value when evaluating the performance of a deep learning model?
Liuji
# How to choose the metric value when evaluating the performance of a deep learning model?

(Asked by: username_0 on 2023-12-23)

When evaluating the performance of a deep learning model (for the purpose of publishing a research work), should we choose the optimal value from the metric curve (such as the accuracy curve) or the convergence value? I think sometimes the optimal value is a result of randomness. So I'm not sure if it can reflect the actual performance of a model. For example, in the accuracy-epoch curve below, should I choose the optimal value (highlighted by the red circle) as the model performance?
[![enter image description here](https://i.sstatic.net/V6P1U.png)](https://i.sstatic.net/V6P1U.png)

**Tags:** machine-learning, deep-learning, metric, testing

> **Comments:**
> **username_1**: Just to clarify, are you asking if you should use the model that corresponds to the optimal metric (e.g. accuracy in this case), right?  
> **username_0**: @username_1 Yes, exactly.  

---

## 1 answers

### Answer by username_2 on 2023-12-23. Score: 2

In short: you should provide the optimal (best) value, but this is not the whole story.

As you said:

> I think sometimes the optimal value is a result of randomness.

This is perfectly true. Therefore, if you want to provide high-quality research results, you *should NOT* consider the single best value due to randomness. You may have been just luck, and got a very high value which, in reality, can be a very rare training run.

What you should do is to *train the model multiple times* according to your computational budget: when the dependence on random numbers is not too strong, even $3$, $5$, or $10$ training runs (with different random seeds) are enough for statistically relevant results. But when randomness is an issue, like in reinforcement learning, way more runs (even 100+) may be necessary.

In this way you can aggregate the result by *mean* (i.e, average expected performance of your model class) and *stddedv* (expected deviation from optimal model.) Doing so, you can obtain much more reliable results.

* Another option is to plot the results with associated *error bars*, such that the wider the bar is the more the variance is.
* A further one, is to get a *p-value* (by comparing the distribution of results) which is interpreted as the probability that method (or algorithm) A beats method B.

(b.t.w. I also assume your training and evaluation setups are mostly, if not entirely, *reproducible* - i.e., deterministic given a fixed random seed.)

As a side fact: is also important to train until your model converges, so until the learning curve stabilizes around a certain value (which is the best one so far, or very close.) This is important to check that both your model and training procedure is healty.


---

