Answers.Body: <p>When formulating a problem in deep learning, we need to come up with a loss function, which uses model weights as parameters. Back-propagation starts at an arbitrary point on the error manifold defined by the loss function and with every iteration intends to move closer to a point that minimises error value by updating the weights. Essentially for every possible set of weights the model can have, there is an associated loss for a given loss function, with our goal being to find the minimum point on this manifold. </p>
  
  <p>Convergence is a term mathematically most common in the study of series and sequences. A model is said to converge when the series <span class="math-container">$s(n) = loss_{w_n}(\hat y, y)$</span> (Where <span class="math-container">$w_n$</span> is the set of weights after the <span class="math-container">$n$</span>'th iteration of back-propagation and <span class="math-container">$s(n)$</span> is the <span class="math-container">$n$</span>'th term of the series) is a converging series. The series is of course an infinite series only if you assume that loss = 0 is never actually achieved, and that learning rate keeps getting smaller. </p>
  
  <p>Essentially meaning, a model converges when its loss actually moves towards a minima (local or global) with a decreasing trend. Its quite rare to actually come across a strictly converging model but convergence is commonly used in a similar manner as convexity is. Strictly speaking rarely exists practically, but is spoken in a manner telling us how close the model is to the ideal scenario for convexity, or in this case convergence. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-08T04:50:21.047Z
Answers.LastActivityDate: 2019-11-08T04:50:21.047Z
Answers.UserDisplayName: ashenoy
Answers.Body: <p>In Machine(or Deep) Learning, <strong>Convergence</strong> means an initial weight moves towards the global minimum of a function by gradient descent as shown below:</p>
  <p><a href="https://i.sstatic.net/Uz1a6rED.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Uz1a6rED.png" alt="enter image description here" /></a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-30T13:07:37.537Z
Answers.LastActivityDate: 2024-07-30T13:07:37.537Z
Answers.UserDisplayName: Super Kai - Kazuya Ito
Body: <p>I came across this <a href="https://www.quora.com/What-is-convergence-in-neural-network" rel="noreferrer">answer on Quora</a>, but it was pretty sparse.  I'm looking for specific meanings in the context of machine learning, but also mathematical and economic notions of the term in general.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-20T16:08:15.953Z
Comments.Text: I would encourage you to limit the scope of this question to just machine learning to avoid making this question being possibly too broad.
Comments.UserDisplayName: nbro
Tags: <neural-networks><machine-learning><terminology><convergence>
Title: What is convergence in machine learning?
UserDisplayName: DukeZhou
ThreadText: # What is convergence in machine learning?
  
  (Asked by: username_0 on 2019-11-08)
  
  I came across this [answer on Quora](https://www.quora.com/What-is-convergence-in-neural-network), but it was pretty sparse. I'm looking for specific meanings in the context of machine learning, but also mathematical and economic notions of the term in general.
  
  **Tags:** neural-networks, machine-learning, terminology, convergence
  > **Comments:**
  > **username_1**: I would encourage you to limit the scope of this question to just machine learning to avoid making this question being possibly too broad.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2019-11-08. Score: 6
  
  When formulating a problem in deep learning, we need to come up with a loss function, which uses model weights as parameters. Back-propagation starts at an arbitrary point on the error manifold defined by the loss function and with every iteration intends to move closer to a point that minimises error value by updating the weights. Essentially for every possible set of weights the model can have, there is an associated loss for a given loss function, with our goal being to find the minimum point on this manifold.
  
  Convergence is a term mathematically most common in the study of series and sequences. A model is said to converge when the series $s(n) = loss\_{w\_n}(\hat y, y)$ (Where $w\_n$ is the set of weights after the $n$'th iteration of back-propagation and $s(n)$ is the $n$'th term of the series) is a converging series. The series is of course an infinite series only if you assume that loss = 0 is never actually achieved, and that learning rate keeps getting smaller.
  
  Essentially meaning, a model converges when its loss actually moves towards a minima (local or global) with a decreasing trend. Its quite rare to actually come across a strictly converging model but convergence is commonly used in a similar manner as convexity is. Strictly speaking rarely exists practically, but is spoken in a manner telling us how close the model is to the ideal scenario for convexity, or in this case convergence.
  
  
  ---
  
  ### Answer by username_3 on 2024-07-30. Score: 0
  
  In Machine(or Deep) Learning, **Convergence** means an initial weight moves towards the global minimum of a function by gradient descent as shown below:
  
  [![enter image description here](https://i.sstatic.net/Uz1a6rED.png)](https://i.sstatic.net/Uz1a6rED.png)
  
  
  ---
  
  