Answers.Body: <p>This article on <a href="https://hackernoon.com/dynamically-expandable-neural-networks-ce75ff2b69cf" rel="nofollow noreferrer">Dynamically Expandable Neural Networks</a> (DEN) (by Harshvardhan Gupta) is based on this paper <a href="https://arxiv.org/pdf/1708.01547v2.pdf" rel="nofollow noreferrer">Lifelong Learning with Dynamically Expandable Networks</a> (by Jeongtae Lee, Jaehong Yoon, Eunho Yang, Sung Ju Hwang)</p>
  
  <p>This presents 3 solutions to increase the capacity of the network if needed retaining whatever useful information from the old model and train the new model:</p>
  
  <ul>
  <li>Selective retraining</li>
  <li>Dynamic Network Expansion </li>
  <li>Network Split/Duplication</li>
  </ul>
  
  <p>To me, it seems that such neural network is dynamic and improving. As such, they answer partially your question. If they don't sorry about that.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-27T11:41:08.590Z
Answers.LastActivityDate: 2019-08-27T23:30:11.123Z
Answers.LastEditDate: 2019-08-27T23:30:11.123Z
Answers.UserDisplayName: luvzfootball
Answers.Body: <p>I mostly studied HMMs and such models are called Infinite HMMs in that specific domain. </p>
  
  <p>I believe that what you are looking for is called Infinite Neural Networks. Not having access to scientific publications, I cannot really refer any work here. However, I found this GitHub repository: <a href="https://github.com/kutoga/going_deeper" rel="nofollow noreferrer">https://github.com/kutoga/going_deeper</a> that provides some implementation and a document with multiple references.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-09-27T13:49:06.163Z
Answers.LastActivityDate: 2018-09-27T13:49:06.163Z
Answers.UserDisplayName: Eskapp
Body: <p>Are there neural networks that can decide to add/delete neurons (or change the neuron models/activation functions or change the assigned meaning for neurons), links or even complete layers during execution time? </p>
  
  <p>I guess that such neural networks overcome the usual separation of learning/inference phases and they continuously live their lives in which learning and self-improving occurs alongside performing inference and actual decision making for which these neural networks were built. Effectively, it could be a neural network that acts as a <a href="http://people.idsia.ch/~juergen/goedelmachine.html" rel="noreferrer">Gödel machine</a>.</p>
  
  <p>I have found the term <em>dynamic neural network</em> but it is connected to adding some delay functions and nothing more.</p>
  
  <p>Of course, such self-improving networks completely redefine the learning strategy, possibly, single shot gradient methods can not be applicable to them. </p>
  
  <p>My question is connected to the neural-symbolic integration, e.g. <a href="https://www.amazon.co.uk/Neural-Symbolic-Cognitive-Reasoning-Technologies/dp/3642092292/" rel="noreferrer"><em>Neural-Symbolic Cognitive Reasoning</em> by Artur S. D'Avila Garcez, 2009</a>. Usually this approach assigns individual neurons to the variables (or groups of neurons to the formula/rule) in the set of formulas in some knowledge base. Of course, if knowledge base expands (e.g. from sensor readings or from inner nonmonotonic inference) then new variables should be added and hence the neural network should be expanded (or contracted) as well.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-09-14T10:36:07.660Z
Comments.Text: as far as my experience takes me, you are looking for genetic algorithms, they keep a population of neural networks and merge them together at random points given how good they do
Comments.UserDisplayName: oren revenge
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-09-14T10:39:12.190Z
Comments.Text: But such external GAs are external controllers which are incapable of learning. I am most interested in self-aware neural networks that themselves can take those decisions.
Comments.UserDisplayName: TomR
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-09-14T12:39:15.413Z
Comments.Text: maybe look at this : https://arxiv.org/ftp/cs/papers/0504/0504055.pdf
Comments.UserDisplayName: Jérémy Blain
Tags: <neural-networks><incremental-learning><online-learning>
Title: Are there dynamic neural networks?
UserDisplayName: TomR
ThreadText: # Are there dynamic neural networks?
  
  (Asked by: username_0 on 2018-09-14)
  
  Are there neural networks that can decide to add/delete neurons (or change the neuron models/activation functions or change the assigned meaning for neurons), links or even complete layers during execution time?
  
  I guess that such neural networks overcome the usual separation of learning/inference phases and they continuously live their lives in which learning and self-improving occurs alongside performing inference and actual decision making for which these neural networks were built. Effectively, it could be a neural network that acts as a [Gödel machine](http://people.idsia.ch/~juergen/goedelmachine.html).
  
  I have found the term *dynamic neural network* but it is connected to adding some delay functions and nothing more.
  
  Of course, such self-improving networks completely redefine the learning strategy, possibly, single shot gradient methods can not be applicable to them.
  
  My question is connected to the neural-symbolic integration, e.g. [*Neural-Symbolic Cognitive Reasoning* by Artur S. D'Avila Garcez, 2009](https://www.amazon.co.uk/Neural-Symbolic-Cognitive-Reasoning-Technologies/dp/3642092292/). Usually this approach assigns individual neurons to the variables (or groups of neurons to the formula/rule) in the set of formulas in some knowledge base. Of course, if knowledge base expands (e.g. from sensor readings or from inner nonmonotonic inference) then new variables should be added and hence the neural network should be expanded (or contracted) as well.
  
  **Tags:** neural-networks, incremental-learning, online-learning
  > **Comments:**
  > **username_1**: as far as my experience takes me, you are looking for genetic algorithms, they keep a population of neural networks and merge them together at random points given how good they do  
  > **username_0**: But such external GAs are external controllers which are incapable of learning. I am most interested in self-aware neural networks that themselves can take those decisions.  
  > **username_2**: maybe look at this : https://arxiv.org/ftp/cs/papers/0504/0504055.pdf  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2019-08-27. Score: 4
  
  This article on [Dynamically Expandable Neural Networks](https://hackernoon.com/dynamically-expandable-neural-networks-ce75ff2b69cf) (DEN) (by Harshvardhan Gupta) is based on this paper [Lifelong Learning with Dynamically Expandable Networks](https://arxiv.org/pdf/1708.01547v2.pdf) (by Jeongtae Lee, Jaehong Yoon, Eunho Yang, Sung Ju Hwang)
  
  This presents 3 solutions to increase the capacity of the network if needed retaining whatever useful information from the old model and train the new model:
  
  * Selective retraining
  * Dynamic Network Expansion
  * Network Split/Duplication
  
  To me, it seems that such neural network is dynamic and improving. As such, they answer partially your question. If they don't sorry about that.
  
  
  ---
  
  ### Answer by username_4 on 2018-09-27. Score: 3
  
  I mostly studied HMMs and such models are called Infinite HMMs in that specific domain.
  
  I believe that what you are looking for is called Infinite Neural Networks. Not having access to scientific publications, I cannot really refer any work here. However, I found this GitHub repository: <https://github.com/kutoga/going_deeper> that provides some implementation and a document with multiple references.
  
  
  ---
  
  