Answers.Body: <p>Consider a function <span class="math-container">$f(x) : \mathcal{R}^m\rightarrow\mathcal{R}^n$</span> defined for <span class="math-container">$x \in X$</span>. If <span class="math-container">$f$</span> is Lipschitz continuous, it has three main properties:</p>
  <ol>
  <li><span class="math-container">$f(x)$</span> is continuous for all <span class="math-container">$x \in X$</span></li>
  <li><span class="math-container">$\frac{d f(x)}{d x}$</span> exists almost everywhere. Meaning, if the derivative is not defined for <span class="math-container">$x \in \mathcal{B}$</span>, where the set <span class="math-container">$\mathcal{B} \subset X$</span>, then <span class="math-container">$\mathcal{B}$</span> has measure zero.</li>
  </ol>
  <ol start="3">
  <li><span class="math-container">$\underset{x \in X}{\sup} \left\lVert \frac{d f(x)}{d x} \right\rVert_2 \leq L$</span>, where <span class="math-container">$L$</span> is the Lipschitz constant, and the norm indicates the <a href="https://en.wikipedia.org/wiki/Matrix_norm#Matrix_norms_induced_by_vector_norms" rel="nofollow noreferrer">induced matrix norm</a> (or if <span class="math-container">$f$</span> is scalar, just the regular 2 norm).</li>
  </ol>
  <p>So it follows that a Lipschitz continuous function is continuous and has a bounded jacobian.</p>
  <p>Now if <span class="math-container">$f$</span> has a lipschitz continuous derivative, then it means <span class="math-container">$\frac{d f}{d x}$</span> is Lipschitz continious, i.e.
  <span class="math-container">\begin{align*}
  \left\lVert \dfrac{d f}{d x}\big|_{x = s} - \dfrac{d f}{d x}\big|_{x = t} \right\rVert_2 \leq M \left\lVert s - t \right\rVert_2 \quad s, t \in X
  \end{align*}</span>
  where <span class="math-container">$M$</span> is the lipschitz constant. So a function with Lipschitz continuous gradient is continuously differentiable and has a bounded hessian.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-22T21:08:07.287Z
Answers.LastActivityDate: 2021-08-22T21:08:07.287Z
Answers.UserDisplayName: Taw
Answers.Body: <p><strong>Answer:</strong>
  A function <span class="math-container">$f$</span> 'Having a Lipschitz continuous derivative' is simply when the derivative of <span class="math-container">$f$</span> is Lipschitz continuous. So it is <strong>not</strong> the derivative of a Lipschitz continuous function.</p>
  <p>Your book states that 'Lipschitz continuous derivatives' as <strong>plural</strong>, meaning that <strong>all</strong> derivatives of <span class="math-container">$f$</span> should be Lipschitz continuous.</p>
  <p><strong>Intuition on Lipschitz continuity:</strong>
  Lipschitz continuity is a property of functions that measures how much the output of a function can change for a given change in the input. A function is Lipschitz continuous if there exists a constant, called the Lipschitz constant, such that the difference between the output of the function for two input points is bounded by the product of the Lipschitz constant and the distance between the two input points.</p>
  <p><span class="math-container">$$d_{Y}(f(x_{1}),f(x_{2}))\leq Kd_{X}(x_{1},x_{2})$$</span>
  where <span class="math-container">$K$</span> is the Lipschitz constant.</p>
  <p>The <a href="https://www.wikiwand.com/en/Lipschitz_continuity" rel="nofollow noreferrer">wiki on Lipschitz continuity</a> gives a nice visualization that might help understand the concept.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-15T12:33:33.167Z
Answers.LastActivityDate: 2023-01-15T12:33:33.167Z
Answers.UserDisplayName: Robin van Hoorn
Body: <p>We can enforce some constraints on functions used in deep learning in order to guarantee optimizations. You can find it in <a href="https://www.deeplearningbook.org/contents/numerical.html" rel="nofollow noreferrer">Numerical Computation</a> of the deep learning book.</p>
  <blockquote>
  <p>In the context of deep learning, we sometimes gain some guarantees by
  restricting ourselves to functions that are either <strong>Lipschitz
  continuous</strong> or have <strong>Lipschitz continuous derivatives</strong>.</p>
  </blockquote>
  <p>They include</p>
  <ol>
  <li>Lipschitz continuous functions</li>
  <li>Having Lipschitz continuous derivatives</li>
  </ol>
  <p>The definition given for Lipschitz continuous function is as follows</p>
  <blockquote>
  <p>A <strong>Lipschitz continuous</strong> function is a function <span class="math-container">$f$</span> whose rate of
  change is bounded by a Lipschitz constant <span class="math-container">$\mathcal{L}$</span>:</p>
  <p><span class="math-container">$$\forall x, \forall y, |f(x)-f(y)| \le \mathcal{L} \|x-y\|_2 $$</span></p>
  </blockquote>
  <p>Now, what is meant by <strong>having Lipschitz continuous derivatives</strong>?</p>
  <p>Does they refer to the derivatives of Lipschitz continuous functions?  If yes, then why do they mention it as a separate option?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-01-20T02:26:40.107Z
Comments.Text: Lipschitzness doesn't imply differentiability, e.g. $f(x)=|x|$ is Lipschitz but not differentiable at $0$. On the other hand, having a Lipschitz continuous derivative doesn't imply Lipschitz continuity, e.g. $f(x)=(-1)^{\mathbb{1}_{x\ge 0}}x^2$. So they're separated.
Comments.UserDisplayName: Tran Khanh
Tags: <deep-learning><math><derivative>
Title: What does it mean "having Lipschitz continuous derivatives"?
UserDisplayName: hanugm
ThreadText: # What does it mean "having Lipschitz continuous derivatives"?
  
  (Asked by: username_0 on 2021-08-22)
  
  We can enforce some constraints on functions used in deep learning in order to guarantee optimizations. You can find it in [Numerical Computation](https://www.deeplearningbook.org/contents/numerical.html) of the deep learning book.
  
  > In the context of deep learning, we sometimes gain some guarantees by
  > restricting ourselves to functions that are either **Lipschitz
  > continuous** or have **Lipschitz continuous derivatives**.
  
  They include
  
  1. Lipschitz continuous functions
  2. Having Lipschitz continuous derivatives
  
  The definition given for Lipschitz continuous function is as follows
  
  > A **Lipschitz continuous** function is a function $f$ whose rate of
  > change is bounded by a Lipschitz constant $\mathcal{L}$:
  >
  > $$\forall x, \forall y, |f(x)-f(y)| \le \mathcal{L} \|x-y\|\_2 $$
  
  Now, what is meant by **having Lipschitz continuous derivatives**?
  
  Does they refer to the derivatives of Lipschitz continuous functions? If yes, then why do they mention it as a separate option?
  
  **Tags:** deep-learning, math, derivative
  > **Comments:**
  > **username_1**: Lipschitzness doesn't imply differentiability, e.g. $f(x)=|x|$ is Lipschitz but not differentiable at $0$. On the other hand, having a Lipschitz continuous derivative doesn't imply Lipschitz continuity, e.g. $f(x)=(-1)^{\mathbb{1}_{x\ge 0}}x^2$. So they're separated.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2021-08-22. Score: 1
  
  Consider a function $f(x) : \mathcal{R}^m\rightarrow\mathcal{R}^n$ defined for $x \in X$. If $f$ is Lipschitz continuous, it has three main properties:
  
  1. $f(x)$ is continuous for all $x \in X$
  2. $\frac{d f(x)}{d x}$ exists almost everywhere. Meaning, if the derivative is not defined for $x \in \mathcal{B}$, where the set $\mathcal{B} \subset X$, then $\mathcal{B}$ has measure zero.
  
  3. $\underset{x \in X}{\sup} \left\lVert \frac{d f(x)}{d x} \right\rVert\_2 \leq L$, where $L$ is the Lipschitz constant, and the norm indicates the [induced matrix norm](https://en.wikipedia.org/wiki/Matrix_norm#Matrix_norms_induced_by_vector_norms) (or if $f$ is scalar, just the regular 2 norm).
  
  So it follows that a Lipschitz continuous function is continuous and has a bounded jacobian.
  
  Now if $f$ has a lipschitz continuous derivative, then it means $\frac{d f}{d x}$ is Lipschitz continious, i.e.
  \begin{align\*}
  \left\lVert \dfrac{d f}{d x}\big|\_{x = s} - \dfrac{d f}{d x}\big|\_{x = t} \right\rVert\_2 \leq M \left\lVert s - t \right\rVert\_2 \quad s, t \in X
  \end{align\*}
  where $M$ is the lipschitz constant. So a function with Lipschitz continuous gradient is continuously differentiable and has a bounded hessian.
  
  
  ---
  
  ### Answer by username_3 on 2023-01-15. Score: 0
  
  **Answer:**
  A function $f$ 'Having a Lipschitz continuous derivative' is simply when the derivative of $f$ is Lipschitz continuous. So it is **not** the derivative of a Lipschitz continuous function.
  
  Your book states that 'Lipschitz continuous derivatives' as **plural**, meaning that **all** derivatives of $f$ should be Lipschitz continuous.
  
  **Intuition on Lipschitz continuity:**
  Lipschitz continuity is a property of functions that measures how much the output of a function can change for a given change in the input. A function is Lipschitz continuous if there exists a constant, called the Lipschitz constant, such that the difference between the output of the function for two input points is bounded by the product of the Lipschitz constant and the distance between the two input points.
  
  $$d\_{Y}(f(x\_{1}),f(x\_{2}))\leq Kd\_{X}(x\_{1},x\_{2})$$
  where $K$ is the Lipschitz constant.
  
  The [wiki on Lipschitz continuity](https://www.wikiwand.com/en/Lipschitz_continuity) gives a nice visualization that might help understand the concept.
  
  
  ---
  
  