Answers.Body: <p><a href="https://ai.stackexchange.com/a/11494/2444">This answer</a> points at some of the more modern approaches. This has been around for a long time in the form of <a href="http://nn.cs.utexas.edu/downloads/papers/stanley.ec02.pdf" rel="nofollow noreferrer">NeAT: Neuroevolution of Augmenting Topologies</a>, originally described in Kenneth Stanley's 2002 paper.</p>
  
  <p>NeAT is available as a package for many languages, including <a href="https://neat-python.readthedocs.io/en/latest/neat_overview.html" rel="nofollow noreferrer">Python</a>, <a href="http://nn.cs.utexas.edu/?jneat" rel="nofollow noreferrer">Java</a>, and <a href="http://nn.cs.utexas.edu/keyword?neat-c" rel="nofollow noreferrer">C++</a>. The algorithm works as a form of <a href="https://en.wikipedia.org/wiki/Genetic_programming" rel="nofollow noreferrer">genetic programming</a>. A population of networks is generated with simple, random, topologies. Then they are evaluated according to a loss function for a specific task. The poorly performing networks are discarded, and the better performing ones are intermixed to generate new variations. This process is iterated until the user wishes it to stop, and typically results in gradual improvement of average population performance against the loss function.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-28T23:54:14.170Z
Answers.Text: The OP might be looking for something like NEAT, but it is worth pointing out that NEAT is not "neural networks editing neural networks" as directly requested.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-29T00:08:55.593Z
Answers.Text: @NeilSlater That's a good point. Perhaps this is more like a GAN in some sense.
Answers.UserDisplayName: John Doucette
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-29T08:39:05.657Z
Answers.Text: It is hard to tell what the OP is aiming at. I think you could probably have something a lot like NEAT, but where a NN is used to guide mutations by learning a better heuristic than random mutation, and that might count. From OP's other question, I am guessing they are aiming for a "strange loop" a la Hofstadter, or some kind of bootstrap and feedback process.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-09T14:08:04.067Z
Answers.Text: does NEAT change the topology as well as the weights? As not sure how one would evaluate the network if it was simply the topology against a given goal
Answers.UserDisplayName: benbyford
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-09T15:04:17.677Z
Answers.Text: @benbyford Yes, it changes both.
Answers.UserDisplayName: John Doucette
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-02T12:38:22.637Z
Answers.Text: @NeilSlater while this is correct in regards to NEAT, hyperneat and es-hyperneat both evolve cppns which encode, meaning it sets connections, weights, and nodes (eshyperneat only for the latter) of a substrate (set of coordinates) as an rnn or cnn so in that case it is literally a nn editing a nn, and John like gans this uses a cppn. but instead of finding patterns in data it finds patterns in nn structure and weights and does so in a non-adversarial way.
Answers.UserDisplayName: nickw
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-28T23:51:32.793Z
Answers.LastActivityDate: 2019-07-28T19:57:16.013Z
Answers.LastEditDate: 2019-07-28T19:57:16.013Z
Answers.UserDisplayName: John Doucette
Answers.Body: <p>Neil Slater is correct when saying that NEAT itself is not neural networks evolving neural networks, what I believe is the closest framework to what the question is asking would be HyperNEAT <a href="http://axon.cs.byu.edu/~dan/778/papers/NeuroEvolution/stanley3" rel="nofollow noreferrer">http://axon.cs.byu.edu/~dan/778/papers/NeuroEvolution/stanley3</a>**.pdf
  HyperNEAT operates in a very similar way to what you are describing, from a ten thousand foot view the algorithm is as follows: 
  1)You lay out nodes for a rnn in a Cartesian space, 2d, 3d, whatever you dimension you wish, this set of coordinates is called the substrate.
  2)A cppn is queried by passing in two coordinates at a time as input, which gives the cppn a search space of a hypercube in 2x the dimension the coordinates are in (for substrates in space > 2d this is very large)
  3)The output of the cppn is used to encode connection, weights, biases, of the rnn coordinates
  4)Then the rnn is evaluated by your fitness function and but the evolution (speciation, reproduction, etc) is ran on the cppn that encoded the rnn. So you evolve a population of cppn "genotypes" that encode rnn or cnn "phenotypes".</p>
  
  <p>The third iteration of NEAT is ES-Hyperneat where all you need to layout in the substrate is the input and output layers (Hyperneat you must layout all hidden nodes of the substrate statically). It uses a subdivision tree to subdivide the search spaces and query the subdivided root coordinates of this tree with the cppn just like hyperneat, checking variance along the way to decide if the new node is in a "high information" topological space, to "evolve" hidden nodes into the substrate (rnn). </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-29T14:04:49.180Z
Answers.LastActivityDate: 2019-07-29T14:26:14.290Z
Answers.LastEditDate: 2019-07-29T14:26:14.290Z
Answers.UserDisplayName: nickw
Answers.Body: <p>Yes, this is an active area of research as we speak. Both using classic algorithms (decision trees, random forests, Bayesian ensembles) as well as neural networks. This can also be done via evolutionary algorithms. I have personally used them for hyperparameter tuning in a few cases where squeezing out a couple of extra points of accuracy was key. </p>
  
  <p>This is in fact what Google is doing with their AutoML system. They are using neural networks for architecture search. </p>
  
  <p>Here is a Github repo with some interesting papers and links on the topic you are describing: <a href="https://github.com/hibayesian/awesome-automl-papers" rel="nofollow noreferrer">https://github.com/hibayesian/awesome-automl-papers</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-28T09:16:46.827Z
Answers.LastActivityDate: 2019-07-07T22:32:18.870Z
Answers.LastEditDate: 2019-07-07T22:32:18.870Z
Answers.UserDisplayName: hisairnessag3
Body: <p>Can neural networks change or evolve other neural networks? Also, could evolutionary algorithms be applied to evolve neural networks?</p>
  
  <p>For example, suppose that we have neural networks A and B. The neural network B changes the neural network A. If B "successfully" changed it, NN A will survive.</p>
  
Tags: <neural-networks><evolutionary-algorithms><agi><neuroevolution>
Title: Can neural networks evolve other neural networks?
UserDisplayName: Dimer
ThreadText: # Can neural networks evolve other neural networks?
  
  (Asked by: username_0 on 2019-03-27)
  
  Can neural networks change or evolve other neural networks? Also, could evolutionary algorithms be applied to evolve neural networks?
  
  For example, suppose that we have neural networks A and B. The neural network B changes the neural network A. If B "successfully" changed it, NN A will survive.
  
  **Tags:** neural-networks, evolutionary-algorithms, agi, neuroevolution
  
  ---
  
  ## 3 answers
  
  ### Answer by username_5 on 2019-03-28. Score: 5 | Accepted answer
  
  Yes, this is an active area of research as we speak. Both using classic algorithms (decision trees, random forests, Bayesian ensembles) as well as neural networks. This can also be done via evolutionary algorithms. I have personally used them for hyperparameter tuning in a few cases where squeezing out a couple of extra points of accuracy was key.
  
  This is in fact what Google is doing with their AutoML system. They are using neural networks for architecture search.
  
  Here is a Github repo with some interesting papers and links on the topic you are describing: <https://github.com/hibayesian/awesome-automl-papers>.
  
  
  ---
  
  ### Answer by username_1 on 2019-03-28. Score: 3
  
  [This answer](https://ai.stackexchange.com/a/11494/2444) points at some of the more modern approaches. This has been around for a long time in the form of [NeAT: Neuroevolution of Augmenting Topologies](http://nn.cs.utexas.edu/downloads/papers/stanley.ec02.pdf), originally described in Kenneth Stanley's 2002 paper.
  
  NeAT is available as a package for many languages, including [Python](https://neat-python.readthedocs.io/en/latest/neat_overview.html), [Java](http://nn.cs.utexas.edu/?jneat), and [C++](http://nn.cs.utexas.edu/keyword?neat-c). The algorithm works as a form of [genetic programming](https://en.wikipedia.org/wiki/Genetic_programming). A population of networks is generated with simple, random, topologies. Then they are evaluated according to a loss function for a specific task. The poorly performing networks are discarded, and the better performing ones are intermixed to generate new variations. This process is iterated until the user wishes it to stop, and typically results in gradual improvement of average population performance against the loss function.
  
  > **Comments:**
  > **username_2**: The OP might be looking for something like NEAT, but it is worth pointing out that NEAT is not "neural networks editing neural networks" as directly requested.  
  > **username_1**: @NeilSlater That's a good point. Perhaps this is more like a GAN in some sense.  
  > **username_2**: It is hard to tell what the OP is aiming at. I think you could probably have something a lot like NEAT, but where a NN is used to guide mutations by learning a better heuristic than random mutation, and that might count. From OP's other question, I am guessing they are aiming for a "strange loop" a la Hofstadter, or some kind of bootstrap and feedback process.  
  > **username_3**: does NEAT change the topology as well as the weights? As not sure how one would evaluate the network if it was simply the topology against a given goal  
  > **username_1**: @benbyford Yes, it changes both.  
  > **username_4**: @NeilSlater while this is correct in regards to NEAT, hyperneat and es-hyperneat both evolve cppns which encode, meaning it sets connections, weights, and nodes (eshyperneat only for the latter) of a substrate (set of coordinates) as an rnn or cnn so in that case it is literally a nn editing a nn, and John like gans this uses a cppn. but instead of finding patterns in data it finds patterns in nn structure and weights and does so in a non-adversarial way.  
  
  ---
  
  ### Answer by username_4 on 2019-07-29. Score: 2
  
  Neil Slater is correct when saying that NEAT itself is not neural networks evolving neural networks, what I believe is the closest framework to what the question is asking would be HyperNEAT <http://axon.cs.byu.edu/~dan/778/papers/NeuroEvolution/stanley3>\*\*.pdf
  HyperNEAT operates in a very similar way to what you are describing, from a ten thousand foot view the algorithm is as follows:
  1)You lay out nodes for a rnn in a Cartesian space, 2d, 3d, whatever you dimension you wish, this set of coordinates is called the substrate.
  2)A cppn is queried by passing in two coordinates at a time as input, which gives the cppn a search space of a hypercube in 2x the dimension the coordinates are in (for substrates in space > 2d this is very large)
  3)The output of the cppn is used to encode connection, weights, biases, of the rnn coordinates
  4)Then the rnn is evaluated by your fitness function and but the evolution (speciation, reproduction, etc) is ran on the cppn that encoded the rnn. So you evolve a population of cppn "genotypes" that encode rnn or cnn "phenotypes".
  
  The third iteration of NEAT is ES-Hyperneat where all you need to layout in the substrate is the input and output layers (Hyperneat you must layout all hidden nodes of the substrate statically). It uses a subdivision tree to subdivide the search spaces and query the subdivided root coordinates of this tree with the cppn just like hyperneat, checking variance along the way to decide if the new node is in a "high information" topological space, to "evolve" hidden nodes into the substrate (rnn).
  
  
  ---
  
  