Answers.Body: <p>I assume you are talking about the training loss. I don't think there's much in the literature, because it's not a big problem by itself. You can pick any reasonable epsilon based on your initial experiments, and you'll likely do fine.</p>
  <p>A bigger problem is overfitting (as the picture below shows). I.e. you can reach very low training loss values, or even zero in some cases, but your model will do poorly for new data.</p>
  <p>A better way is to track the <a href="https://stats.stackexchange.com/a/474148/280286"><em>validation</em> loss</a> during your training, and stop training when it bottomed out, or stopped improving.</p>
  <p><a href="https://1.cms.s81c.com/sites/default/files/2021-03-03/classic%20overfitting_0.jpg" rel="nofollow noreferrer"><img src="https://1.cms.s81c.com/sites/default/files/2021-03-03/classic%20overfitting_0.jpg" alt="enter image description here" /></a></p>
  <p>Source: <a href="https://1.cms.s81c.com/sites/default/files/2021-03-03/classic%20overfitting_0.jpg" rel="nofollow noreferrer">https://1.cms.s81c.com/sites/default/files/2021-03-03/classic%20overfitting_0.jpg</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-17T21:10:21.517Z
Answers.LastActivityDate: 2023-05-17T21:10:21.517Z
Answers.UserDisplayName: Alex
Body: <p>In deep learning, one way to determine whether the training has converged is to observe the movement of the loss values over iterations or epochs. One can choose any <span class="math-container">$\epsilon$</span> threshold and any metric. If the value is less than <span class="math-container">$\epsilon$</span>, then the training has converged.</p>
  <p>My question is: how big is the <span class="math-container">$\epsilon$</span> value that is usually used? Are there examples of papers that specifically state the threshold?</p>
  
Tags: <deep-learning><convergence>
Title: How big is the threshold that is usually used in determining the convergence of loss values in deep learning?
UserDisplayName: poglhar
ThreadText: # How big is the threshold that is usually used in determining the convergence of loss values in deep learning?
  
  (Asked by: username_0 on 2023-05-17)
  
  In deep learning, one way to determine whether the training has converged is to observe the movement of the loss values over iterations or epochs. One can choose any $\epsilon$ threshold and any metric. If the value is less than $\epsilon$, then the training has converged.
  
  My question is: how big is the $\epsilon$ value that is usually used? Are there examples of papers that specifically state the threshold?
  
  **Tags:** deep-learning, convergence
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-05-17. Score: 0
  
  I assume you are talking about the training loss. I don't think there's much in the literature, because it's not a big problem by itself. You can pick any reasonable epsilon based on your initial experiments, and you'll likely do fine.
  
  A bigger problem is overfitting (as the picture below shows). I.e. you can reach very low training loss values, or even zero in some cases, but your model will do poorly for new data.
  
  A better way is to track the [*validation* loss](https://stats.stackexchange.com/a/474148/280286) during your training, and stop training when it bottomed out, or stopped improving.
  
  [![enter image description here](https://1.cms.s81c.com/sites/default/files/2021-03-03/classic%20overfitting_0.jpg)](https://1.cms.s81c.com/sites/default/files/2021-03-03/classic%20overfitting_0.jpg)
  
  Source: <https://1.cms.s81c.com/sites/default/files/2021-03-03/classic%20overfitting_0.jpg>
  
  
  ---
  
  