Answers.Body: <p><strong>Neuroevolution Through Augmenting Topologies</strong> or <strong>NEAT</strong> may be what you are referring to. The original paper by Kenneth O. Stanley is <a href="http://nn.cs.utexas.edu/downloads/papers/stanley.ec02.pdf" rel="noreferrer">here</a></p>
  
  <p>NEAT combines a neural network and a genetic algorithm. Instead of using back propagation or gradient descent to "train" your network, NEAT creates a population of very simple neural networks (no connections) and evolves them with <strong>fitness evaluation, crossover,</strong> and <strong>mutation.</strong> The genome syntax: every connection gene has a few settings. In node, Out node, Weight of connection, activated, and innovation. <strong>In, Out,</strong> and <strong>Weight</strong> values are the same as regular neural networks. <strong>Enabled</strong> and <strong>Disabled</strong> genes are well, enabled and disabled. The innovation value is possibly the most defining feature of NEAT, since it allows for crossover of different topologies and historical tracking of each connection.<a href="https://i.sstatic.net/J9zMq.png" rel="noreferrer"><img src="https://i.sstatic.net/J9zMq.png" alt="enter image description here"></a>NEAT can mutate or change both its weights and connections, so for example, <code>Parent1</code> and <code>Parent2</code> has 5 of the same connections, represented by innovation / ID numbers 1 through 5. Since they have the same connection nodes, the genetic algorithm will randomly pick either <code>Parent1 weight</code> or <code>Parent2 weight</code>. The excess and disjoint genes are inherited from the more fit parent.<a href="https://i.sstatic.net/l0je3.png" rel="noreferrer"><img src="https://i.sstatic.net/l0je3.png" alt="NEAT crossover"></a> NEAT will then mutate each genome, shown in the image below.<a href="https://i.sstatic.net/DbJ29.png" rel="noreferrer"><img src="https://i.sstatic.net/DbJ29.png" alt="enter image description here"></a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-13T13:36:27.783Z
Answers.LastActivityDate: 2019-08-13T13:36:27.783Z
Answers.UserDisplayName: Terry T.
Answers.Body: <p>I haven't read any relevant paper about this, but I have seen some implementations based on what you are describing, arbitrarily called <strong>DGNN (Dynamic Growing Neural Network)</strong>. </p>
  
  <p>Hope this term can help your search.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-13T08:35:37.343Z
Answers.LastActivityDate: 2019-08-13T16:39:40.450Z
Answers.LastEditDate: 2019-08-13T16:39:40.450Z
Answers.UserDisplayName: Loheek
Answers.Body: <p>At <a href="https://arxiv.org/abs/1711.09667" rel="nofollow noreferrer">DeepChess</a> they trained stacked autoencoders:</p>
  <blockquote>
  <p><strong>Training Pos2Vec</strong>: We first trained a deep belief network (DBN), which would later
  serve as the initial weights for supervised training. The DBN is based
  on stacked autoencoders which are trained using layer-wise
  unsupervised training. The network consists of five fully connected
  layers of sizes: 773–600–400–200–100. We initially trained the first
  layer (i.e., a 3-layer (773–600–773) autoencoder), before fixing its
  weights and training the weights of a new (600–400–600) autoencoder,
  and so on.</p>
  </blockquote>
  <p>I suppose this matches your criteria, although the network's architecture isn't very flexible. They don't introduce skip-connections etc.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-01T10:08:47.520Z
Answers.LastActivityDate: 2022-07-01T10:08:47.520Z
Answers.UserDisplayName: NikoNyrh
Body: <p>For an experiment that I'm working on, I want to train a deep network in a special way. I want to initialize and train a small network first, then, in a specific way, I want to increase network depth leading to a bigger network which is subsequently to be trained. This process will be repeated until one reaches the desired depth.</p>
  <p>It would be great if anybody heard of anything similar and could point out to me some related work. I think in some paper I read something about a related technique where people used something similar, but I don't find it anymore.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-08-13T15:55:07.093Z
Comments.Text: it is common when dealing with images to train this way using progressive resizing
Comments.UserDisplayName: juvian
Tags: <neural-networks><deep-learning><reference-request><training><neuroevolution>
Title: Iteratively and adaptively increasing the network size during training
UserDisplayName: Leroy Od
ThreadText: # Iteratively and adaptively increasing the network size during training
  
  (Asked by: username_0 on 2019-08-13)
  
  For an experiment that I'm working on, I want to train a deep network in a special way. I want to initialize and train a small network first, then, in a specific way, I want to increase network depth leading to a bigger network which is subsequently to be trained. This process will be repeated until one reaches the desired depth.
  
  It would be great if anybody heard of anything similar and could point out to me some related work. I think in some paper I read something about a related technique where people used something similar, but I don't find it anymore.
  
  **Tags:** neural-networks, deep-learning, reference-request, training, neuroevolution
  > **Comments:**
  > **username_1**: it is common when dealing with images to train this way using progressive resizing  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_2 on 2019-08-13. Score: 5
  
  **Neuroevolution Through Augmenting Topologies** or **NEAT** may be what you are referring to. The original paper by Kenneth O. Stanley is [here](http://nn.cs.utexas.edu/downloads/papers/stanley.ec02.pdf)
  
  NEAT combines a neural network and a genetic algorithm. Instead of using back propagation or gradient descent to "train" your network, NEAT creates a population of very simple neural networks (no connections) and evolves them with **fitness evaluation, crossover,** and **mutation.** The genome syntax: every connection gene has a few settings. In node, Out node, Weight of connection, activated, and innovation. **In, Out,** and **Weight** values are the same as regular neural networks. **Enabled** and **Disabled** genes are well, enabled and disabled. The innovation value is possibly the most defining feature of NEAT, since it allows for crossover of different topologies and historical tracking of each connection.[![enter image description here](https://i.sstatic.net/J9zMq.png)](https://i.sstatic.net/J9zMq.png)NEAT can mutate or change both its weights and connections, so for example, `Parent1` and `Parent2` has 5 of the same connections, represented by innovation / ID numbers 1 through 5. Since they have the same connection nodes, the genetic algorithm will randomly pick either `Parent1 weight` or `Parent2 weight`. The excess and disjoint genes are inherited from the more fit parent.[![NEAT crossover](https://i.sstatic.net/l0je3.png)](https://i.sstatic.net/l0je3.png) NEAT will then mutate each genome, shown in the image below.[![enter image description here](https://i.sstatic.net/DbJ29.png)](https://i.sstatic.net/DbJ29.png)
  
  
  ---
  
  ### Answer by username_3 on 2019-08-13. Score: 2
  
  I haven't read any relevant paper about this, but I have seen some implementations based on what you are describing, arbitrarily called **DGNN (Dynamic Growing Neural Network)**.
  
  Hope this term can help your search.
  
  
  ---
  
  ### Answer by username_4 on 2022-07-01. Score: 0
  
  At [DeepChess](https://arxiv.org/abs/1711.09667) they trained stacked autoencoders:
  
  > **Training Pos2Vec**: We first trained a deep belief network (DBN), which would later
  > serve as the initial weights for supervised training. The DBN is based
  > on stacked autoencoders which are trained using layer-wise
  > unsupervised training. The network consists of five fully connected
  > layers of sizes: 773–600–400–200–100. We initially trained the first
  > layer (i.e., a 3-layer (773–600–773) autoencoder), before fixing its
  > weights and training the weights of a new (600–400–600) autoencoder,
  > and so on.
  
  I suppose this matches your criteria, although the network's architecture isn't very flexible. They don't introduce skip-connections etc.
  
  
  ---
  
  