Answers.Body: <p>You could say that NAS fits into the domain of Meta Learning or Meta Machine learning.</p>
  
  <p>I've pulled the NAS papers from my notes, this is a collection of papers/lectures that I personally found very interesting. It's sorted in rough chronological descending order, and <code>***</code> means influential / must read.</p>
  
  <p>Quoc V. Le and Barret Zoph are to good authors on the topic.</p>
  
  <ul>
  <li><a href="https://arxiv.org/pdf/1901.11117v1.pdf" rel="noreferrer">The Evolved Transformer</a></li>
  <li><a href="https://arxiv.org/pdf/1904.01569.pdf" rel="noreferrer">Exploring Randomly Wired Neural Networks for Image Recognition</a></li>
  <li><a href="https://arxiv.org/pdf/1810.05749v1.pdf" rel="noreferrer">GRAPH HYPERNETWORKS FOR NEURAL ARCHITECTURE SEARCH</a></li>
  <li><a href="https://arxiv.org/pdf/1808.02822v1.pdf" rel="noreferrer">Backprop Evolution</a></li>
  <li><a href="https://arxiv.org/pdf/1712.00559v3.pdf" rel="noreferrer">Progressive Neural Architecture Search</a></li>
  <li>*** <a href="https://arxiv.org/pdf/1806.09055v1.pdf" rel="noreferrer">DARTS: Differentiable Architecture Search</a></li>
  <li>*** <a href="https://arxiv.org/pdf/1802.03268.pdf" rel="noreferrer">Efficient Neural Architecture Search via Parameter Sharing - ENAS</a></li>
  <li>*** <a href="https://arxiv.org/pdf/1712.00559v2.pdf" rel="noreferrer">Progressive Neural Architecture Search</a></li>
  <li><a href="https://arxiv.org/pdf/1805.07440v1.pdf" rel="noreferrer">AlphaX: eXploring Neural Architectures with Deep Neural Networks and Monte Carlo Tree Search</a></li>
  <li><a href="https://www.youtube.com/watch?v=OR-IKyP4ZpI" rel="noreferrer">Automatic Machine Learning - Prof. Frank Hutter</a></li>
  <li><a href="https://www.youtube.com/watch?v=sROrvtXnT7Q" rel="noreferrer">Google Brain - Neural Architecture Search - Quoc Le</a></li>
  <li>*** <a href="https://arxiv.org/pdf/1802.01548.pdf" rel="noreferrer">Regularized Evolution for Image Classifier Architecture Search</a></li>
  <li><a href="https://arxiv.org/pdf/1803.00684v1.pdf" rel="noreferrer">Autostacker: A Compositional Evolutionary Learning System</a></li>
  <li><a href="https://arxiv.org/pdf/1801.01952v1.pdf" rel="noreferrer">Generating Neural Networks with Neural Networks</a></li>
  <li><a href="https://arxiv.org/pdf/1712.07420.pdf" rel="noreferrer">Finding Competitive Network Architectures Within a Day Using UCT</a></li>
  <li><a href="https://www.oreilly.com/ideas/neuroevolution-a-different-kind-of-deep-learning" rel="noreferrer">Neuroevolution: A different kind of deep learning</a></li>
  <li><a href="https://arxiv.org/pdf/1703.00548.pdf" rel="noreferrer">Evolving Deep Neural Networks</a></li>
  <li><a href="https://www.youtube.com/watch?v=TERCdog1ddE" rel="noreferrer">Pieter Abbeel: Deep Learning-to-Learn Robotic Control</a></li>
  <li>*** <a href="https://arxiv.org/pdf/1708.05344.pdf" rel="noreferrer">SMASH: One-Shot Model Architecture Search through HyperNetworks</a></li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-20T21:17:26.960Z
Answers.LastActivityDate: 2019-05-20T21:17:26.960Z
Answers.UserDisplayName: Benjamin Crouzier
Answers.Body: <p>Here are two review articles:</p>
  
  <ul>
  <li><p>Elsken, Metzen, Hutter: <a href="https://arxiv.org/abs/1808.05377" rel="nofollow noreferrer">Neural Architecture Search: A Survey</a> (2019), Journal of Machine Learning Research 20, 1-21 </p></li>
  <li><p>He, Zhao, Chu: <a href="https://arxiv.org/abs/1908.00709" rel="nofollow noreferrer">AutoML: A Survey of the State-of-the-Art</a> (2019) </p></li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-11T14:17:27.707Z
Answers.Text: The question is not about AutoML, but only NAS, so I suggest you remove your second suggestion.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-11T15:48:00.317Z
Answers.Text: @nbro 1. about half of the paper is about NAS. 2. if there were tons of reviews one could get finicky, but losing one out of two?
Answers.UserDisplayName: Mathieu Bouville
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-11T15:51:01.277Z
Answers.Text: I don't get what you mean by "but losing one out of two" in this context.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-11T12:34:16.817Z
Answers.LastActivityDate: 2019-11-11T14:16:45.350Z
Answers.LastEditDate: 2019-11-11T14:16:45.350Z
Answers.UserDisplayName: Mathieu Bouville
Answers.Body: <p><a href="https://en.wikipedia.org/wiki/Neural_architecture_search" rel="nofollow noreferrer">Neural architecture search (NAS)</a> is a method of automating the design (that is, the choice of the values of the hyper-parameters) of artificial neural networks. There are different approaches to search the space of neural network architectures. For example, you can use reinforcement learning or evolutionary (or genetic) algorithms.</p>
  
  <p>Check out the paper <a href="https://arxiv.org/pdf/1611.01578.pdf" rel="nofollow noreferrer">Neural Architecture Search with Reinforcement Learning</a> (2017), by Barret Zoph and Quoc V. Le, where the authors train, using reinforcement learning (specifically, <a href="http://www-anw.cs.umass.edu/~barto/courses/cs687/williams92simple.pdf" rel="nofollow noreferrer">REINFORCE</a>), a recurrent neural network (the "controller") to generate (convolutional and recurrent) neural network architectures, so that to maximise the expected accuracy of the generated architectures on a validation dataset. They achieve some good results using this approach.</p>
  
  <p>See also <a href="https://arxiv.org/pdf/1802.03268.pdf" rel="nofollow noreferrer">Efficient Neural Architecture Search via Parameter Sharing</a> (2018), by Hieu Pham, Melody Y. Guan, Barret Zoph, Quoc V. Le and Jeff Dean (which thus includes some of the authors of <a href="https://arxiv.org/pdf/1611.01578.pdf" rel="nofollow noreferrer">NAS</a>), which is similar to <a href="https://arxiv.org/pdf/1611.01578.pdf" rel="nofollow noreferrer">NAS</a>, but more efficient, hence the acronym ENAS (efficient NAS).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-20T17:05:54.080Z
Answers.LastActivityDate: 2019-07-18T23:28:03.637Z
Answers.LastEditDate: 2019-07-18T23:28:03.637Z
Answers.UserDisplayName: nbro
Body: <p>I have come across something that IBM offers called <em>neural architecture search</em>. You feed it a data set and it outputs an initial neural architecture that you can train.</p>
  <p>How is <em>neural architecture search</em> (NAS) performed? Do they use heuristics, or is this meta machine learning?</p>
  <p>If you have any papers on NAS, I would appreciate if you can provide a link to them.</p>
  
Tags: <neural-networks><reference-request><neuroevolution><hyperparameter-optimization><neural-architecture-search>
Title: How is neural architecture search performed?
UserDisplayName: Adam Geringer
ThreadText: # How is neural architecture search performed?
  
  (Asked by: username_0 on 2019-05-20)
  
  I have come across something that IBM offers called *neural architecture search*. You feed it a data set and it outputs an initial neural architecture that you can train.
  
  How is *neural architecture search* (NAS) performed? Do they use heuristics, or is this meta machine learning?
  
  If you have any papers on NAS, I would appreciate if you can provide a link to them.
  
  **Tags:** neural-networks, reference-request, neuroevolution, hyperparameter-optimization, neural-architecture-search
  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_1 on 2019-05-20. Score: 6
  
  You could say that NAS fits into the domain of Meta Learning or Meta Machine learning.
  
  I've pulled the NAS papers from my notes, this is a collection of papers/lectures that I personally found very interesting. It's sorted in rough chronological descending order, and `***` means influential / must read.
  
  Quoc V. Le and Barret Zoph are to good authors on the topic.
  
  * [The Evolved Transformer](https://arxiv.org/pdf/1901.11117v1.pdf)
  * [Exploring Randomly Wired Neural Networks for Image Recognition](https://arxiv.org/pdf/1904.01569.pdf)
  * [GRAPH HYPERNETWORKS FOR NEURAL ARCHITECTURE SEARCH](https://arxiv.org/pdf/1810.05749v1.pdf)
  * [Backprop Evolution](https://arxiv.org/pdf/1808.02822v1.pdf)
  * [Progressive Neural Architecture Search](https://arxiv.org/pdf/1712.00559v3.pdf)
  * \*\*\* [DARTS: Differentiable Architecture Search](https://arxiv.org/pdf/1806.09055v1.pdf)
  * \*\*\* [Efficient Neural Architecture Search via Parameter Sharing - ENAS](https://arxiv.org/pdf/1802.03268.pdf)
  * \*\*\* [Progressive Neural Architecture Search](https://arxiv.org/pdf/1712.00559v2.pdf)
  * [AlphaX: eXploring Neural Architectures with Deep Neural Networks and Monte Carlo Tree Search](https://arxiv.org/pdf/1805.07440v1.pdf)
  * [Automatic Machine Learning - Prof. Frank Hutter](https://www.youtube.com/watch?v=OR-IKyP4ZpI)
  * [Google Brain - Neural Architecture Search - Quoc Le](https://www.youtube.com/watch?v=sROrvtXnT7Q)
  * \*\*\* [Regularized Evolution for Image Classifier Architecture Search](https://arxiv.org/pdf/1802.01548.pdf)
  * [Autostacker: A Compositional Evolutionary Learning System](https://arxiv.org/pdf/1803.00684v1.pdf)
  * [Generating Neural Networks with Neural Networks](https://arxiv.org/pdf/1801.01952v1.pdf)
  * [Finding Competitive Network Architectures Within a Day Using UCT](https://arxiv.org/pdf/1712.07420.pdf)
  * [Neuroevolution: A different kind of deep learning](https://www.oreilly.com/ideas/neuroevolution-a-different-kind-of-deep-learning)
  * [Evolving Deep Neural Networks](https://arxiv.org/pdf/1703.00548.pdf)
  * [Pieter Abbeel: Deep Learning-to-Learn Robotic Control](https://www.youtube.com/watch?v=TERCdog1ddE)
  * \*\*\* [SMASH: One-Shot Model Architecture Search through HyperNetworks](https://arxiv.org/pdf/1708.05344.pdf)
  
  
  ---
  
  ### Answer by username_2 on 2019-11-11. Score: 1
  
  Here are two review articles:
  
  * Elsken, Metzen, Hutter: [Neural Architecture Search: A Survey](https://arxiv.org/abs/1808.05377) (2019), Journal of Machine Learning Research 20, 1-21
  * He, Zhao, Chu: [AutoML: A Survey of the State-of-the-Art](https://arxiv.org/abs/1908.00709) (2019)
  
  > **Comments:**
  > **username_3**: The question is not about AutoML, but only NAS, so I suggest you remove your second suggestion.  
  > **username_2**: @username_3 1. about half of the paper is about NAS. 2. if there were tons of reviews one could get finicky, but losing one out of two?  
  > **username_3**: I don't get what you mean by "but losing one out of two" in this context.  
  
  ---
  
  ### Answer by username_3 on 2019-05-20. Score: 0
  
  [Neural architecture search (NAS)](https://en.wikipedia.org/wiki/Neural_architecture_search) is a method of automating the design (that is, the choice of the values of the hyper-parameters) of artificial neural networks. There are different approaches to search the space of neural network architectures. For example, you can use reinforcement learning or evolutionary (or genetic) algorithms.
  
  Check out the paper [Neural Architecture Search with Reinforcement Learning](https://arxiv.org/pdf/1611.01578.pdf) (2017), by Barret Zoph and Quoc V. Le, where the authors train, using reinforcement learning (specifically, [REINFORCE](http://www-anw.cs.umass.edu/~barto/courses/cs687/williams92simple.pdf)), a recurrent neural network (the "controller") to generate (convolutional and recurrent) neural network architectures, so that to maximise the expected accuracy of the generated architectures on a validation dataset. They achieve some good results using this approach.
  
  See also [Efficient Neural Architecture Search via Parameter Sharing](https://arxiv.org/pdf/1802.03268.pdf) (2018), by Hieu Pham, Melody Y. Guan, Barret Zoph, Quoc V. Le and Jeff Dean (which thus includes some of the authors of [NAS](https://arxiv.org/pdf/1611.01578.pdf)), which is similar to [NAS](https://arxiv.org/pdf/1611.01578.pdf), but more efficient, hence the acronym ENAS (efficient NAS).
  
  
  ---
  
  