Answers.Body: <p>The derivative <span class="math-container">$f'(x)$</span> is correlated with <span class="math-container">$f(x)$</span> <em>in a certain sense</em>. In fact, <span class="math-container">$f'(x)$</span> is a function of <span class="math-container">$f$</span>, so we could even say that there's a cause-effect relationship.</p>
  
  <p>The derivative at a specific point <span class="math-container">$c$</span> of the domain, i.e. <span class="math-container">$f'(c)$</span>, can either be negative or positive. If <span class="math-container">$f'(c) &gt; 0$</span>, then <span class="math-container">$f(c)$</span> is increasing (with respect to an increase of <span class="math-container">$x$</span>). If <span class="math-container">$f'(c) &lt; 0$</span>, then <span class="math-container">$f(c)$</span> decreasing  (with respect to an increase of <span class="math-container">$x$</span>). </p>
  
  <p>This can easily be seen from an example. Consider <span class="math-container">$f(x) = x^2$</span>, then <span class="math-container">$f'(x) = 2x$</span>. Let <span class="math-container">$c = 2$</span>, then <span class="math-container">$f'(2) = 4$</span>, so the function is increasing. In fact, <span class="math-container">$f(1) = 2 \leq f(2) = 4 \leq f(3) = 9$</span>. Similarly, let <span class="math-container">$c = -1$</span>, then <span class="math-container">$f'(-1) = -2$</span>, so the function is decreasing. In fact, <span class="math-container">$\leq f(-2) = 4 \geq f(-1) = 1 \geq  f(0) = 0$</span> (note that the function is decreasing as we increase <span class="math-container">$x$</span>!). </p>
  
  <p>Consider a model with only one parameter, then the partial derivative of the loss function with respect to that parameter corresponds to the derivative of the loss function. So, the reasoning above applies to this model. What about a model with more than one parameter? The same thing happens. </p>
  
  <p><em>If the function decreases, does its derivative also decrease?</em> In general, no, and this can easily be seen from a plot of a function and its derivative. For example, consider a plot of a parabola and its derivative (which is a linear function).</p>
  
  <p><a href="https://i.sstatic.net/6xmWu.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/6xmWu.png" alt="enter image description here"></a></p>
  
  <p>On the left of the y-axis, the parabola is decreasing, but its derivative is increasing, while, on the right of the y-axis, the parabola is increasing and the linear function is still increasing.</p>
  
  <p>This is the same thing with a loss function of an ML model and its partial derivative.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-07T01:44:55.347Z
Answers.Text: @DuttaA The function that computes $f'$ needs to have $f$ as input. This function is the [differential operator](https://en.wikipedia.org/wiki/Differential_operator). I am not completely sure about the mathematical formalism, but this should give you the idea and info to further investigate it!
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-06T18:45:39.673Z
Answers.LastActivityDate: 2020-04-06T18:45:39.673Z
Answers.UserDisplayName: nbro
Body: <p>I am wondering about the correlation between the loss and the derivative of the loss wrt a single scalar parameter, with the same sample. That means: considering a machine learning model with parameters <span class="math-container">$\theta \in R$</span>, I want to figure out the relationship between <span class="math-container">$Loss(x)$</span> and <span class="math-container">$\frac{\partial Loss(x)}{\partial \theta_i}$</span>, where <span class="math-container">$i \in \{1,2,3,...,n\}$</span></p>
  
  <p>Intuitively, I would like to consider that they are in a positive correlation, is it right? If it is right, how can I prove it in a mathematical way? </p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-06T16:25:56.530Z
Comments.Text: You need a function relating $x$ and $\theta$, otherwise your situation is not fully described. And typically in supervised learning loss is related to a comparison of transformed $x$ with a ground truth $y$ (although not always required)
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-06T16:32:07.800Z
Comments.Text: For instance, for linear regression, you might have $\hat{y} = \theta \cdot x + b$ and $\mathcal{L}(\hat{y}, y) = \frac{1}{2}(\hat{y} - y)^2$ where $y$ is ground truth. For a concrete example you should [edit] in how $\theta$ is being used with $x$, and maybe the loss function you are using too. If you want a generic description/proof then you still need to specify how the loss function, $x$ and $\theta$ relate before someone could say whether correlation is always positive or not
Comments.UserDisplayName: Neil Slater
Tags: <machine-learning><deep-learning><math><proofs>
Title: Is the derivative of the loss wrt a single scalar parameter proportional to the loss?
UserDisplayName: Zyuuu
ThreadText: # Is the derivative of the loss wrt a single scalar parameter proportional to the loss?
  
  (Asked by: username_0 on 2020-04-03)
  
  I am wondering about the correlation between the loss and the derivative of the loss wrt a single scalar parameter, with the same sample. That means: considering a machine learning model with parameters $\theta \in R$, I want to figure out the relationship between $Loss(x)$ and $\frac{\partial Loss(x)}{\partial \theta\_i}$, where $i \in \{1,2,3,...,n\}$
  
  Intuitively, I would like to consider that they are in a positive correlation, is it right? If it is right, how can I prove it in a mathematical way?
  
  **Tags:** machine-learning, deep-learning, math, proofs
  > **Comments:**
  > **username_1**: You need a function relating $x$ and $\theta$, otherwise your situation is not fully described. And typically in supervised learning loss is related to a comparison of transformed $x$ with a ground truth $y$ (although not always required)  
  > **username_1**: For instance, for linear regression, you might have $\hat{y} = \theta \cdot x + b$ and $\mathcal{L}(\hat{y}, y) = \frac{1}{2}(\hat{y} - y)^2$ where $y$ is ground truth. For a concrete example you should [edit] in how $\theta$ is being used with $x$, and maybe the loss function you are using too. If you want a generic description/proof then you still need to specify how the loss function, $x$ and $\theta$ relate before someone could say whether correlation is always positive or not  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-04-06. Score: 1
  
  The derivative $f'(x)$ is correlated with $f(x)$ *in a certain sense*. In fact, $f'(x)$ is a function of $f$, so we could even say that there's a cause-effect relationship.
  
  The derivative at a specific point $c$ of the domain, i.e. $f'(c)$, can either be negative or positive. If $f'(c) > 0$, then $f(c)$ is increasing (with respect to an increase of $x$). If $f'(c) < 0$, then $f(c)$ decreasing (with respect to an increase of $x$).
  
  This can easily be seen from an example. Consider $f(x) = x^2$, then $f'(x) = 2x$. Let $c = 2$, then $f'(2) = 4$, so the function is increasing. In fact, $f(1) = 2 \leq f(2) = 4 \leq f(3) = 9$. Similarly, let $c = -1$, then $f'(-1) = -2$, so the function is decreasing. In fact, $\leq f(-2) = 4 \geq f(-1) = 1 \geq f(0) = 0$ (note that the function is decreasing as we increase $x$!).
  
  Consider a model with only one parameter, then the partial derivative of the loss function with respect to that parameter corresponds to the derivative of the loss function. So, the reasoning above applies to this model. What about a model with more than one parameter? The same thing happens.
  
  *If the function decreases, does its derivative also decrease?* In general, no, and this can easily be seen from a plot of a function and its derivative. For example, consider a plot of a parabola and its derivative (which is a linear function).
  
  [![enter image description here](https://i.sstatic.net/6xmWu.png)](https://i.sstatic.net/6xmWu.png)
  
  On the left of the y-axis, the parabola is decreasing, but its derivative is increasing, while, on the right of the y-axis, the parabola is increasing and the linear function is still increasing.
  
  This is the same thing with a loss function of an ML model and its partial derivative.
  
  > **Comments:**
  > **username_2**: @DuttaA The function that computes $f'$ needs to have $f$ as input. This function is the [differential operator](https://en.wikipedia.org/wiki/Differential_operator). I am not completely sure about the mathematical formalism, but this should give you the idea and info to further investigate it!  
  
  ---
  
  