Answers.Body: <p>I'm not an expert in this area, but it would appear to depend on the choice of activation function:</p>
  
  <ul>
  <li><span class="math-container">$e^x$</span> is not Lipschitz continuous. See <a href="https://en.wikipedia.org/wiki/Lipschitz_continuity" rel="nofollow noreferrer">Analytic functions which are not Lipschitz continuous</a>.</li>
  <li><span class="math-container">$\tanh(x)$</span> <a href="https://books.google.co.uk/books?id=Sd0cCAAAQBAJ&amp;pg=PA222&amp;lpg=PA222&amp;dq=tanh%20lipschitz%20continuous&amp;source=bl&amp;ots=xgIQAQYDEw&amp;sig=plaHUKkiPYB388sT2ht-iro6ntY&amp;hl=en&amp;sa=X&amp;ved=0ahUKEwiLvfyo14TPAhVmCsAKHYxXBoMQ6AEIQTAG#v=onepage&amp;q=tanh%20lipschitz%20continuous&amp;f=false" rel="nofollow noreferrer">is</a>.</li>
  </ul>
  
  <p>That said, <a href="http://web.mit.edu/~esontag/public_html/FTP_DIR/00cdc-papers-refs-eds/CD001925.PDF" rel="nofollow noreferrer">this paper</a> appears to give some conditions (specifically for <em>dynamic</em> ANNs) for which networks with activation function involving <span class="math-container">$e^x$</span> <em>can</em> be Lipschitz continuous, so possibly the above is not the whole story.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2016-09-10T11:27:02.597Z
Answers.LastActivityDate: 2018-12-28T19:35:12.083Z
Answers.LastEditDate: 2018-12-28T19:35:12.083Z
Answers.UserDisplayName: NietzscheanAI
Body: <p>My question is regarding standard dense-connected feed forward neural networks with sigmoidal activation.</p>
  
  <p>I am studying Bayesian Optimization for hyper-parameter selection for neural networks. There is no doubt that this is an effective method, but I just wan't to delve a little deeper into the maths.</p>
  
  <p><strong>Question:</strong> Are neural networks <a href="http://mathworld.wolfram.com/LipschitzFunction.html" rel="nofollow">Lipschitz</a> functions?</p>
  
Tags: <neural-networks><optimization><math>
Title: Are FFNN (MLP) Lipschitz functions?
UserDisplayName: Faur
ThreadText: # Are FFNN (MLP) Lipschitz functions?
  
  (Asked by: username_0 on 2016-09-10)
  
  My question is regarding standard dense-connected feed forward neural networks with sigmoidal activation.
  
  I am studying Bayesian Optimization for hyper-parameter selection for neural networks. There is no doubt that this is an effective method, but I just wan't to delve a little deeper into the maths.
  
  **Question:** Are neural networks [Lipschitz](http://mathworld.wolfram.com/LipschitzFunction.html) functions?
  
  **Tags:** neural-networks, optimization, math
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2016-09-10. Score: 2 | Accepted answer
  
  I'm not an expert in this area, but it would appear to depend on the choice of activation function:
  
  * $e^x$ is not Lipschitz continuous. See [Analytic functions which are not Lipschitz continuous](https://en.wikipedia.org/wiki/Lipschitz_continuity).
  * $\tanh(x)$ [is](https://books.google.co.uk/books?id=Sd0cCAAAQBAJ&pg=PA222&lpg=PA222&dq=tanh%20lipschitz%20continuous&source=bl&ots=xgIQAQYDEw&sig=plaHUKkiPYB388sT2ht-iro6ntY&hl=en&sa=X&ved=0ahUKEwiLvfyo14TPAhVmCsAKHYxXBoMQ6AEIQTAG#v=onepage&q=tanh%20lipschitz%20continuous&f=false).
  
  That said, [this paper](http://web.mit.edu/~esontag/public_html/FTP_DIR/00cdc-papers-refs-eds/CD001925.PDF) appears to give some conditions (specifically for *dynamic* ANNs) for which networks with activation function involving $e^x$ *can* be Lipschitz continuous, so possibly the above is not the whole story.
  
  
  ---
  
  