title: Continuous vs Discrete artificial neural networks
body: <p>I realize that this is probably a very niche question, but has anyone had experience with working with continuous neural networks? I'm specifically interested in what a continuous neural network may be useful for vs what you normally use discrete neural networks for.</p>
  <p>For clarity I will clear up what I mean by continuous neural network as I suppose it can be interpreted to mean different things. I do <strong>not</strong> mean that the activation function is continuous. Rather I allude to the idea of a increasing the number of  neurons in the hidden layer to an infinite amount.</p>
  <p>So for clarity, here is the architecture of your typical discreet NN:
  <a href="https://i.stack.imgur.com/XU442.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/XU442.png" alt="alt text" /></a><br />
  <sub>(source: <a href="https://sites.google.com/site/garamatt/nn.png" rel="nofollow noreferrer">garamatt at sites.google.com</a>)</sub></p>
  <p>The <code>x</code> are the input, the <code>g</code> is the activation of the hidden layer, the <code>v</code> are the weights of the hidden layer, the <code>w</code> are the weights of the output layer, the <code>b</code> is the bias and apparently the output layer has a linear activation (namely none.)</p>
  <p>The difference between a discrete NN and a continuous NN is depicted by this figure:
  <a href="https://i.stack.imgur.com/kl3uA.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/kl3uA.png" alt="alt text" /></a><br />
  <sub>(source: <a href="https://sites.google.com/site/garamatt/nn2.png" rel="nofollow noreferrer">garamatt at sites.google.com</a>)</sub></p>
  <p>That is you let the number of hidden neurons become infinite so that your final output is an integral. In practice this means that instead of computing a deterministic sum you instead must approximate the corresponding integral with quadrature.</p>
  <p>Apparently its a common misconception with neural networks that too many hidden neurons produces over-fitting.</p>
  <p>My question is specifically, given this definition of discrete and continuous neural networks, I was wondering if anyone had experience working with the latter and what sort of things they used them for.</p>
  <p>Further description on the topic can be found here:
  <a href="http://www.iro.umontreal.ca/%7Elisa/seminaires/18-04-2006.pdf" rel="nofollow noreferrer">http://www.iro.umontreal.ca/~lisa/seminaires/18-04-2006.pdf</a></p>