<p>It is just an assumption. If we assume <span class="math-container">$\epsilon$</span> is small enough (depending on the function <span class="math-container">$f$</span>), you can remove the limit <span class="math-container">$\lim_{\epsilon \to 0}$</span> for the approximation.</p>

CC BY-SA 4.0
2021-08-17T09:24:05.907Z
2021-08-17T09:24:05.907Z
OmG
<p>The equation
<span class="math-container">$$f(x + \epsilon) \approx f(x) + \epsilon f'(x)$$</span>
is justified from taylor series. It is not derived from the limit definition of a derivative. Let <span class="math-container">$f'(a)$</span> and <span class="math-container">$f''(a)$</span> exist for <span class="math-container">$a$</span> in the interval <span class="math-container">$(x, x+\epsilon)$</span>. Then,
<span class="math-container">\begin{align*} 
f(x + \epsilon) = f(x) + \epsilon f'(x) + \frac{\epsilon^2}{2}f''(b)
\end{align*}</span>
where <span class="math-container">$b$</span> is some number between <span class="math-container">$(x, x + \epsilon)$</span>. If <span class="math-container">$\epsilon &lt;&lt; 1$</span>, then <span class="math-container">$\frac{\epsilon^2}{2}f''(b)$</span> is small so it can be ignored, leading to the approximation.</p>
<p><a href="https://en.wikipedia.org/wiki/Taylor%27s_theorem#Explicit_formulas_for_the_remainder" rel="nofollow noreferrer">taylor's theorem</a></p>

CC BY-SA 4.0
2021-08-17T19:15:55.107Z
2021-08-17T19:15:55.107Z
Taw
<p>Consider the following paragraph from <a href="https://www.deeplearningbook.org/contents/numerical.html" rel="nofollow noreferrer">NUMERICAL COMPUTATION</a> of the deep learning book..</p>
<blockquote>
<p>Suppose we have a function <span class="math-container">$y = f(x)$</span>, where both <span class="math-container">$x$</span> and <span class="math-container">$y$</span> are real
numbers. The derivative of this function is denoted as <span class="math-container">$f'(x)$</span> or as
<span class="math-container">$\dfrac{dy}{dx}$</span>. The derivative <span class="math-container">$f'(x)$</span> gives the slope of <span class="math-container">$f(x)$</span> at
the point <span class="math-container">$x$</span>. In other words, it speciﬁes how to scale a small change
in the input to obtain the corresponding change in the output:
<span class="math-container">$f(x+\epsilon) \approx f(x) + \epsilon f'(x)$</span>.</p>
</blockquote>
<p>I have doubt in the equation <span class="math-container">$f(x+\epsilon) \approx f(x) + \epsilon f'(x)$</span> given in the paragraph.</p>
<blockquote>
<p>In strict sense, the derivative function <span class="math-container">$f'$</span> of a real valued
function <span class="math-container">$f$</span> is defined as</p>
<p><span class="math-container">$$f'(x) = \lim_{\epsilon \rightarrow 0}
 \dfrac{f(x+\epsilon)-f(x)}{\epsilon}$$</span></p>
<p>wherever the limit exists.</p>
</blockquote>
<p>If I replace the original definition of the derivative as follows</p>
<p><span class="math-container">$$f'(x) \approx \dfrac{f(x+\epsilon)-f(x)}{\epsilon}$$</span></p>
<p>then I can obtain the equation given in the paragraph i.e,  <span class="math-container">$f(x+\epsilon) \approx f(x) + \epsilon f'(x)$</span>.</p>
<p>But, my doubt is that how can I modify the definition with <span class="math-container">$\lim\limits_{\epsilon \rightarrow 0}$</span> to an approximation with out limit? How can the following two are same?</p>
<p><span class="math-container">$$f'(x) = \lim_{\epsilon \rightarrow 0} \dfrac{f(x+\epsilon)-f(x)}{\epsilon} \text { and } f'(x) \approx \dfrac{f(x+\epsilon)-f(x)}{\epsilon}$$</span></p>

<math><optimization><gradient-descent><derivative>
Reason for relaxing limit in derivative in this context?
hanugm
# Reason for relaxing limit in derivative in this context?

(Asked by: username_0 on 2021-08-17)

Consider the following paragraph from [NUMERICAL COMPUTATION](https://www.deeplearningbook.org/contents/numerical.html) of the deep learning book..

> Suppose we have a function $y = f(x)$, where both $x$ and $y$ are real
> numbers. The derivative of this function is denoted as $f'(x)$ or as
> $\dfrac{dy}{dx}$. The derivative $f'(x)$ gives the slope of $f(x)$ at
> the point $x$. In other words, it speciﬁes how to scale a small change
> in the input to obtain the corresponding change in the output:
> $f(x+\epsilon) \approx f(x) + \epsilon f'(x)$.

I have doubt in the equation $f(x+\epsilon) \approx f(x) + \epsilon f'(x)$ given in the paragraph.

> In strict sense, the derivative function $f'$ of a real valued
> function $f$ is defined as
>
> $$f'(x) = \lim\_{\epsilon \rightarrow 0}
> \dfrac{f(x+\epsilon)-f(x)}{\epsilon}$$
>
> wherever the limit exists.

If I replace the original definition of the derivative as follows

$$f'(x) \approx \dfrac{f(x+\epsilon)-f(x)}{\epsilon}$$

then I can obtain the equation given in the paragraph i.e, $f(x+\epsilon) \approx f(x) + \epsilon f'(x)$.

But, my doubt is that how can I modify the definition with $\lim\limits\_{\epsilon \rightarrow 0}$ to an approximation with out limit? How can the following two are same?

$$f'(x) = \lim\_{\epsilon \rightarrow 0} \dfrac{f(x+\epsilon)-f(x)}{\epsilon} \text { and } f'(x) \approx \dfrac{f(x+\epsilon)-f(x)}{\epsilon}$$

**Tags:** math, optimization, gradient-descent, derivative


---

## 2 answers

### Answer by username_2 on 2021-08-17. Score: 2 | Accepted answer

The equation
$$f(x + \epsilon) \approx f(x) + \epsilon f'(x)$$
is justified from taylor series. It is not derived from the limit definition of a derivative. Let $f'(a)$ and $f''(a)$ exist for $a$ in the interval $(x, x+\epsilon)$. Then,
\begin{align\*}
f(x + \epsilon) = f(x) + \epsilon f'(x) + \frac{\epsilon^2}{2}f''(b)
\end{align\*}
where $b$ is some number between $(x, x + \epsilon)$. If $\epsilon << 1$, then $\frac{\epsilon^2}{2}f''(b)$ is small so it can be ignored, leading to the approximation.

[taylor's theorem](https://en.wikipedia.org/wiki/Taylor%27s_theorem#Explicit_formulas_for_the_remainder)


---

### Answer by username_1 on 2021-08-17. Score: 1

It is just an assumption. If we assume $\epsilon$ is small enough (depending on the function $f$), you can remove the limit $\lim\_{\epsilon \to 0}$ for the approximation.


---

