Answers.Body: <p>It is just an assumption. If we assume <span class="math-container">$\epsilon$</span> is small enough (depending on the function <span class="math-container">$f$</span>), you can remove the limit <span class="math-container">$\lim_{\epsilon \to 0}$</span> for the approximation.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-17T09:24:05.907Z
Answers.LastActivityDate: 2021-08-17T09:24:05.907Z
Answers.UserDisplayName: OmG
Answers.Body: <p>The equation
  <span class="math-container">$$f(x + \epsilon) \approx f(x) + \epsilon f'(x)$$</span>
  is justified from taylor series. It is not derived from the limit definition of a derivative. Let <span class="math-container">$f'(a)$</span> and <span class="math-container">$f''(a)$</span> exist for <span class="math-container">$a$</span> in the interval <span class="math-container">$(x, x+\epsilon)$</span>. Then,
  <span class="math-container">\begin{align*} 
  f(x + \epsilon) = f(x) + \epsilon f'(x) + \frac{\epsilon^2}{2}f''(b)
  \end{align*}</span>
  where <span class="math-container">$b$</span> is some number between <span class="math-container">$(x, x + \epsilon)$</span>. If <span class="math-container">$\epsilon &lt;&lt; 1$</span>, then <span class="math-container">$\frac{\epsilon^2}{2}f''(b)$</span> is small so it can be ignored, leading to the approximation.</p>
  <p><a href="https://en.wikipedia.org/wiki/Taylor%27s_theorem#Explicit_formulas_for_the_remainder" rel="nofollow noreferrer">taylor's theorem</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-17T19:15:55.107Z
Answers.LastActivityDate: 2021-08-17T19:15:55.107Z
Answers.UserDisplayName: Taw
Body: <p>Consider the following paragraph from <a href="https://www.deeplearningbook.org/contents/numerical.html" rel="nofollow noreferrer">NUMERICAL COMPUTATION</a> of the deep learning book..</p>
  <blockquote>
  <p>Suppose we have a function <span class="math-container">$y = f(x)$</span>, where both <span class="math-container">$x$</span> and <span class="math-container">$y$</span> are real
  numbers. The derivative of this function is denoted as <span class="math-container">$f'(x)$</span> or as
  <span class="math-container">$\dfrac{dy}{dx}$</span>. The derivative <span class="math-container">$f'(x)$</span> gives the slope of <span class="math-container">$f(x)$</span> at
  the point <span class="math-container">$x$</span>. In other words, it speciﬁes how to scale a small change
  in the input to obtain the corresponding change in the output:
  <span class="math-container">$f(x+\epsilon) \approx f(x) + \epsilon f'(x)$</span>.</p>
  </blockquote>
  <p>I have doubt in the equation <span class="math-container">$f(x+\epsilon) \approx f(x) + \epsilon f'(x)$</span> given in the paragraph.</p>
  <blockquote>
  <p>In strict sense, the derivative function <span class="math-container">$f'$</span> of a real valued
  function <span class="math-container">$f$</span> is defined as</p>
  <p><span class="math-container">$$f'(x) = \lim_{\epsilon \rightarrow 0}
   \dfrac{f(x+\epsilon)-f(x)}{\epsilon}$$</span></p>
  <p>wherever the limit exists.</p>
  </blockquote>
  <p>If I replace the original definition of the derivative as follows</p>
  <p><span class="math-container">$$f'(x) \approx \dfrac{f(x+\epsilon)-f(x)}{\epsilon}$$</span></p>
  <p>then I can obtain the equation given in the paragraph i.e,  <span class="math-container">$f(x+\epsilon) \approx f(x) + \epsilon f'(x)$</span>.</p>
  <p>But, my doubt is that how can I modify the definition with <span class="math-container">$\lim\limits_{\epsilon \rightarrow 0}$</span> to an approximation with out limit? How can the following two are same?</p>
  <p><span class="math-container">$$f'(x) = \lim_{\epsilon \rightarrow 0} \dfrac{f(x+\epsilon)-f(x)}{\epsilon} \text { and } f'(x) \approx \dfrac{f(x+\epsilon)-f(x)}{\epsilon}$$</span></p>
  
Tags: <math><optimization><gradient-descent><derivative>
Title: Reason for relaxing limit in derivative in this context?
UserDisplayName: hanugm
ThreadText: # Reason for relaxing limit in derivative in this context?
  
  (Asked by: username_0 on 2021-08-17)
  
  Consider the following paragraph from [NUMERICAL COMPUTATION](https://www.deeplearningbook.org/contents/numerical.html) of the deep learning book..
  
  > Suppose we have a function $y = f(x)$, where both $x$ and $y$ are real
  > numbers. The derivative of this function is denoted as $f'(x)$ or as
  > $\dfrac{dy}{dx}$. The derivative $f'(x)$ gives the slope of $f(x)$ at
  > the point $x$. In other words, it speciﬁes how to scale a small change
  > in the input to obtain the corresponding change in the output:
  > $f(x+\epsilon) \approx f(x) + \epsilon f'(x)$.
  
  I have doubt in the equation $f(x+\epsilon) \approx f(x) + \epsilon f'(x)$ given in the paragraph.
  
  > In strict sense, the derivative function $f'$ of a real valued
  > function $f$ is defined as
  >
  > $$f'(x) = \lim\_{\epsilon \rightarrow 0}
  > \dfrac{f(x+\epsilon)-f(x)}{\epsilon}$$
  >
  > wherever the limit exists.
  
  If I replace the original definition of the derivative as follows
  
  $$f'(x) \approx \dfrac{f(x+\epsilon)-f(x)}{\epsilon}$$
  
  then I can obtain the equation given in the paragraph i.e, $f(x+\epsilon) \approx f(x) + \epsilon f'(x)$.
  
  But, my doubt is that how can I modify the definition with $\lim\limits\_{\epsilon \rightarrow 0}$ to an approximation with out limit? How can the following two are same?
  
  $$f'(x) = \lim\_{\epsilon \rightarrow 0} \dfrac{f(x+\epsilon)-f(x)}{\epsilon} \text { and } f'(x) \approx \dfrac{f(x+\epsilon)-f(x)}{\epsilon}$$
  
  **Tags:** math, optimization, gradient-descent, derivative
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2021-08-17. Score: 2 | Accepted answer
  
  The equation
  $$f(x + \epsilon) \approx f(x) + \epsilon f'(x)$$
  is justified from taylor series. It is not derived from the limit definition of a derivative. Let $f'(a)$ and $f''(a)$ exist for $a$ in the interval $(x, x+\epsilon)$. Then,
  \begin{align\*}
  f(x + \epsilon) = f(x) + \epsilon f'(x) + \frac{\epsilon^2}{2}f''(b)
  \end{align\*}
  where $b$ is some number between $(x, x + \epsilon)$. If $\epsilon << 1$, then $\frac{\epsilon^2}{2}f''(b)$ is small so it can be ignored, leading to the approximation.
  
  [taylor's theorem](https://en.wikipedia.org/wiki/Taylor%27s_theorem#Explicit_formulas_for_the_remainder)
  
  
  ---
  
  ### Answer by username_1 on 2021-08-17. Score: 1
  
  It is just an assumption. If we assume $\epsilon$ is small enough (depending on the function $f$), you can remove the limit $\lim\_{\epsilon \to 0}$ for the approximation.
  
  
  ---
  
  