Answers.Body: <p>I don't think there's a theoretical reason for the shape, it's just showing the diminishing returns as training goes.</p>
  <p>As for a mathematical function that describes this, <a href="https://en.wikipedia.org/wiki/Exponential_decay" rel="nofollow noreferrer">exponential decay</a> may fit well here (in case of loss, or inverse exponential decay if accuracy).</p>
  <p>Also, I would not assume that this abrupt flat line of your examples are a universal.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-17T00:09:54.887Z
Answers.LastActivityDate: 2024-09-17T00:09:54.887Z
Answers.UserDisplayName: talles
Body: <p>In training a neural network, you often see the curve showing how fast the neural network is getting better. It usually grows very fast then slows down to almost horizontal.</p>
  
  <p>Is there a mathematical formula that matches these curves?</p>
  
  <p><a href="https://i.sstatic.net/e8og1.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/e8og1.jpg" alt="enter image description here"></a></p>
  
  <p>Some similar curves are:</p>
  
  <p><span class="math-container">$$y=1-e^{-x}$$</span></p>
  
  <p><span class="math-container">$$y=\frac{x}{1+x}$$</span></p>
  
  <p><span class="math-container">$$y=\tanh(x)$$</span></p>
  
  <p><span class="math-container">$$y=1+x-\sqrt{1+x^2}$$</span></p>
  
  <p>Is there a theoretical reason for this shape?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-09T15:36:34.440Z
Comments.Text: As you mention, "It **usually** grows very fast then slows down", so, of course, there isn't a single mathematical formula that would describe all learning curves. So, I suggest you clarify your question, which seems to be quite interesting.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-09T19:08:39.970Z
Comments.Text: @nbro OK. By usually I mean always. And of course it slows down because the neural network can't keep increasing intelligence forever. But these curves do almost always look the same and do usually keep growing for a long time but very slowly. I'm just wondering if one can theortically model this curve and produce a formula for it.
Comments.UserDisplayName: zooby
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-09T19:11:48.190Z
Comments.Text: Someone changed it to "how fast the nerual network is converging". This is not what the curves show. The curves might show, for example, how the neural network improves in its game of chess.
Comments.UserDisplayName: zooby
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-09T19:24:15.407Z
Comments.Text: I had changed that sentence because you had written something like "network is gaining intelligence", which is a little bit arguable. Maybe "convergence" isn't the most appropriate word and I didn't think much about it. Anyway, learning curves look a bit like logarithmic curves.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-11T04:59:56.883Z
Comments.Text: I don't think there is a universal mathematical formula that describes the learning curve, but what I know is that not all curves look like your diagram. In Q-learning there is almost no overfitting so the curve continue to rise but in other task there maybe overfitting so it may rise then fall slowly again. Maybe one could fit a function to for the curve?
Comments.UserDisplayName: Clement
Tags: <neural-networks><machine-learning><math><learning-curve>
Title: Is there a mathematical formula that describes the learning curve in neural networks?
UserDisplayName: zooby
ThreadText: # Is there a mathematical formula that describes the learning curve in neural networks?
  
  (Asked by: username_0 on 2020-01-09)
  
  In training a neural network, you often see the curve showing how fast the neural network is getting better. It usually grows very fast then slows down to almost horizontal.
  
  Is there a mathematical formula that matches these curves?
  
  [![enter image description here](https://i.sstatic.net/e8og1.jpg)](https://i.sstatic.net/e8og1.jpg)
  
  Some similar curves are:
  
  $$y=1-e^{-x}$$
  
  $$y=\frac{x}{1+x}$$
  
  $$y=\tanh(x)$$
  
  $$y=1+x-\sqrt{1+x^2}$$
  
  Is there a theoretical reason for this shape?
  
  **Tags:** neural-networks, machine-learning, math, learning-curve
  > **Comments:**
  > **username_1**: As you mention, "It **usually** grows very fast then slows down", so, of course, there isn't a single mathematical formula that would describe all learning curves. So, I suggest you clarify your question, which seems to be quite interesting.  
  > **username_0**: @nbro OK. By usually I mean always. And of course it slows down because the neural network can't keep increasing intelligence forever. But these curves do almost always look the same and do usually keep growing for a long time but very slowly. I'm just wondering if one can theortically model this curve and produce a formula for it.  
  > **username_0**: Someone changed it to "how fast the nerual network is converging". This is not what the curves show. The curves might show, for example, how the neural network improves in its game of chess.  
  > **username_1**: I had changed that sentence because you had written something like "network is gaining intelligence", which is a little bit arguable. Maybe "convergence" isn't the most appropriate word and I didn't think much about it. Anyway, learning curves look a bit like logarithmic curves.  
  > **username_2**: I don't think there is a universal mathematical formula that describes the learning curve, but what I know is that not all curves look like your diagram. In Q-learning there is almost no overfitting so the curve continue to rise but in other task there maybe overfitting so it may rise then fall slowly again. Maybe one could fit a function to for the curve?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2024-09-17. Score: 0
  
  I don't think there's a theoretical reason for the shape, it's just showing the diminishing returns as training goes.
  
  As for a mathematical function that describes this, [exponential decay](https://en.wikipedia.org/wiki/Exponential_decay) may fit well here (in case of loss, or inverse exponential decay if accuracy).
  
  Also, I would not assume that this abrupt flat line of your examples are a universal.
  
  
  ---
  
  