<p>I'm not an expert in this area, but it would appear to depend on the choice of activation function:</p>

<ul>
<li><span class="math-container">$e^x$</span> is not Lipschitz continuous. See <a href="https://en.wikipedia.org/wiki/Lipschitz_continuity" rel="nofollow noreferrer">Analytic functions which are not Lipschitz continuous</a>.</li>
<li><span class="math-container">$\tanh(x)$</span> <a href="https://books.google.co.uk/books?id=Sd0cCAAAQBAJ&amp;pg=PA222&amp;lpg=PA222&amp;dq=tanh%20lipschitz%20continuous&amp;source=bl&amp;ots=xgIQAQYDEw&amp;sig=plaHUKkiPYB388sT2ht-iro6ntY&amp;hl=en&amp;sa=X&amp;ved=0ahUKEwiLvfyo14TPAhVmCsAKHYxXBoMQ6AEIQTAG#v=onepage&amp;q=tanh%20lipschitz%20continuous&amp;f=false" rel="nofollow noreferrer">is</a>.</li>
</ul>

<p>That said, <a href="http://web.mit.edu/~esontag/public_html/FTP_DIR/00cdc-papers-refs-eds/CD001925.PDF" rel="nofollow noreferrer">this paper</a> appears to give some conditions (specifically for <em>dynamic</em> ANNs) for which networks with activation function involving <span class="math-container">$e^x$</span> <em>can</em> be Lipschitz continuous, so possibly the above is not the whole story.</p>

CC BY-SA 4.0
2016-09-10T11:27:02.597Z
2018-12-28T19:35:12.083Z
2018-12-28T19:35:12.083Z
NietzscheanAI
<p>My question is regarding standard dense-connected feed forward neural networks with sigmoidal activation.</p>

<p>I am studying Bayesian Optimization for hyper-parameter selection for neural networks. There is no doubt that this is an effective method, but I just wan't to delve a little deeper into the maths.</p>

<p><strong>Question:</strong> Are neural networks <a href="http://mathworld.wolfram.com/LipschitzFunction.html" rel="nofollow">Lipschitz</a> functions?</p>

<neural-networks><optimization><math>
Are FFNN (MLP) Lipschitz functions?
Faur
# Are FFNN (MLP) Lipschitz functions?

(Asked by: username_0 on 2016-09-10)

My question is regarding standard dense-connected feed forward neural networks with sigmoidal activation.

I am studying Bayesian Optimization for hyper-parameter selection for neural networks. There is no doubt that this is an effective method, but I just wan't to delve a little deeper into the maths.

**Question:** Are neural networks [Lipschitz](http://mathworld.wolfram.com/LipschitzFunction.html) functions?

**Tags:** neural-networks, optimization, math


---

## 1 answers

### Answer by username_1 on 2016-09-10. Score: 2 | Accepted answer

I'm not an expert in this area, but it would appear to depend on the choice of activation function:

* $e^x$ is not Lipschitz continuous. See [Analytic functions which are not Lipschitz continuous](https://en.wikipedia.org/wiki/Lipschitz_continuity).
* $\tanh(x)$ [is](https://books.google.co.uk/books?id=Sd0cCAAAQBAJ&pg=PA222&lpg=PA222&dq=tanh%20lipschitz%20continuous&source=bl&ots=xgIQAQYDEw&sig=plaHUKkiPYB388sT2ht-iro6ntY&hl=en&sa=X&ved=0ahUKEwiLvfyo14TPAhVmCsAKHYxXBoMQ6AEIQTAG#v=onepage&q=tanh%20lipschitz%20continuous&f=false).

That said, [this paper](http://web.mit.edu/~esontag/public_html/FTP_DIR/00cdc-papers-refs-eds/CD001925.PDF) appears to give some conditions (specifically for *dynamic* ANNs) for which networks with activation function involving $e^x$ *can* be Lipschitz continuous, so possibly the above is not the whole story.


---

