Answers.Body: <p>In general <span class="math-container">$|f(x) - f_k(x)| \leq \epsilon$</span> doesn't ensure <span class="math-container">$|\nabla f(x) - \nabla f_k(x)| \leq c\epsilon$</span>. And for neural networks there is no reason to believe it will happen either. </p>
  
  <p>You can also look at Sobolev Training Paper (<a href="https://arxiv.org/abs/1706.04859" rel="nofollow noreferrer">https://arxiv.org/abs/1706.04859</a>). In particular, note that Sobolev training was better than critic training, which indirectly may indicate approximating function may not be the same as approximating gradient and function. In Sobolev training, the network is trained to match gradient and function whereas in critic training network is trained to match function. They produce quite different results which might give us some hints about the above problem. </p>
  
  <p>In general, if two functions are arbitrary close, they might not be close in gradients.</p>
  
  <p>Edit: 
  (Trying to come up with a negative example)
  Consider <span class="math-container">$f(x) = g(x) + \epsilon \sin (\frac {kx} {\epsilon}) $</span>. <span class="math-container">$g(x)$</span> is some neural network. Now, we train some another neural network <span class="math-container">$h(x)$</span> to fit <span class="math-container">$f(x)$</span> and after training we get <span class="math-container">$h(x) = g(x)$</span> (<span class="math-container">$h(x)$</span> and <span class="math-container">$g(x)$</span> have same weights precisely). However, <span class="math-container">$\nabla f_x =\nabla g_x + k\cos (\frac {kx} {\epsilon})$</span> is not arbitarariy close <span class="math-container">$\nabla g_x$</span>.  </p>
  
  <p>I hope this example is enough to prove that a neural network that nicely approximates the function may not nicely approximate the gradients and no such result can be proved mathematically rigorously. However, considering the paper in the discussion, you might think for practical purposes it works. However, if you have both function and grad information available that is expected to work better. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-06T05:15:18.907Z
Answers.Text: great answer! i wonder - would regular gradient descent avoid this problem?
Answers.UserDisplayName: stats_noob
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-09-22T04:05:27.047Z
Answers.LastActivityDate: 2018-09-22T17:43:31.713Z
Answers.LastEditDate: 2018-09-22T17:43:31.713Z
Answers.UserDisplayName: Umang Gupta
Body: <p>Let’s say I have a neural net doing classification and I’m doing stochastic gradient descent to train it. If I know that my current approximation is a decent approximation, can I conclude that my gradient is a decent approximation of the gradient of the true classifier everywhere?</p>
  
  <p>Specifically, suppose that I have a true loss function, $f$, and an estimation of it, $f_k$. Is it the case that there exists a $c$ (dependent on $f_k$) such that for all $x$ and $\epsilon &gt; 0$ if $|f(x)-f_k(x)|&lt;\epsilon$ then $|\nabla f(x) - \nabla f_k(x)|&lt;c\epsilon$? This isn’t true for general functions, but it may be true for neural nets. If this exact statement isn’t true, is there something along these lines that is? What if we place some restrictions on the NN?</p>
  
  <p>The goal I have in mind is that I’m trying to figure out how to calculate how long I can use a particular sample to estimate the gradient without the error getting too bad. If I am in a context where resampling is costly, it may be worth reusing the same sample many times as long as I’m not making my error too large. My long-term goal is to come up with a bound on how much error I have if I use the same sample $k$ times, which doesn’t seem to be something in the literature as far as I’ve found.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-02-06T05:15:37.790Z
Comments.Text: i wonder - would regular gradient descent avoid this problem?
Comments.UserDisplayName: stats_noob
Tags: <neural-networks><gradient-descent>
Title: Do good approximations produce good gradients?
UserDisplayName: Stella Biderman
ThreadText: # Do good approximations produce good gradients?
  
  (Asked by: username_0 on 2018-09-21)
  
  Let’s say I have a neural net doing classification and I’m doing stochastic gradient descent to train it. If I know that my current approximation is a decent approximation, can I conclude that my gradient is a decent approximation of the gradient of the true classifier everywhere?
  
  Specifically, suppose that I have a true loss function, $f$, and an estimation of it, $f\_k$. Is it the case that there exists a $c$ (dependent on $f\_k$) such that for all $x$ and $\epsilon > 0$ if $|f(x)-f\_k(x)|<\epsilon$ then $|\nabla f(x) - \nabla f\_k(x)|
  
  The goal I have in mind is that I’m trying to figure out how to calculate how long I can use a particular sample to estimate the gradient without the error getting too bad. If I am in a context where resampling is costly, it may be worth reusing the same sample many times as long as I’m not making my error too large. My long-term goal is to come up with a bound on how much error I have if I use the same sample $k$ times, which doesn’t seem to be something in the literature as far as I’ve found.
  
  **Tags:** neural-networks, gradient-descent
  > **Comments:**
  > **username_1**: i wonder - would regular gradient descent avoid this problem?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2018-09-22. Score: 5 | Accepted answer
  
  In general $|f(x) - f\_k(x)| \leq \epsilon$ doesn't ensure $|\nabla f(x) - \nabla f\_k(x)| \leq c\epsilon$. And for neural networks there is no reason to believe it will happen either.
  
  You can also look at Sobolev Training Paper (<https://arxiv.org/abs/1706.04859>). In particular, note that Sobolev training was better than critic training, which indirectly may indicate approximating function may not be the same as approximating gradient and function. In Sobolev training, the network is trained to match gradient and function whereas in critic training network is trained to match function. They produce quite different results which might give us some hints about the above problem.
  
  In general, if two functions are arbitrary close, they might not be close in gradients.
  
  Edit:
  (Trying to come up with a negative example)
  Consider $f(x) = g(x) + \epsilon \sin (\frac {kx} {\epsilon}) $. $g(x)$ is some neural network. Now, we train some another neural network $h(x)$ to fit $f(x)$ and after training we get $h(x) = g(x)$ ($h(x)$ and $g(x)$ have same weights precisely). However, $\nabla f\_x =\nabla g\_x + k\cos (\frac {kx} {\epsilon})$ is not arbitarariy close $\nabla g\_x$.
  
  I hope this example is enough to prove that a neural network that nicely approximates the function may not nicely approximate the gradients and no such result can be proved mathematically rigorously. However, considering the paper in the discussion, you might think for practical purposes it works. However, if you have both function and grad information available that is expected to work better.
  
  > **Comments:**
  > **username_1**: great answer! i wonder - would regular gradient descent avoid this problem?  
  
  ---
  
  