Answers.Body: <p>(I will repeat a few details that you're already aware of, so that other users can also understand the context).</p>
  
  <p>In the <a href="https://arxiv.org/pdf/1611.01578.pdf" rel="nofollow noreferrer">Neural Architecture Search (NAS) paper</a> (that I mention <a href="https://ai.stackexchange.com/a/12435/2444">in my answer</a> to the question you link to in your question), the agent is the controller (see also this question <a href="https://ai.stackexchange.com/q/13119/2444">Is there any difference between a control and an action in reinforcement learning?</a>), which is implemented as a recurrent neural network (RNN). This controller produces actions (or controls), which are strings that represent the hyper-parameters of a neural network (see e.g. section 3.2), based on the reward that it receives, which, in this case, is the accuracy on the validation dataset that the designed (by the controller) and trained neural network obtains. </p>
  
  <p>In this context, the controller is thus the reinforcement learning agent (or policy). The controller is an RNN that is represented by a vector of parameters, <span class="math-container">$\boldsymbol{\theta}_\text{c}$</span>. This controller is trained using a policy gradient method to maximize the expected accuracy on the validation dataset (see e.g. section 3 of the paper). So <span class="math-container">$\boldsymbol{\theta}_\text{c}$</span> are adjusted using this policy gradient method so that the controller generates NN architectures that, after trained on some task, produce higher accuracy on a validation dataset.</p>
  
  <p>The objective function of the controller is thus </p>
  
  <p><span class="math-container">$$
  J(\boldsymbol{\theta}_\text{c}) = \mathbb{E}[R]
  $$</span></p>
  
  <p>where <span class="math-container">$R$</span> is the accuracy on the validation dataset. Why <em>expected</em>? You will be performing this operation multiple times, so, intuitively, you want an average of the accuracy that you obtain on the validation dataset using multiple neural network architectures that the controller might produce.</p>
  
  <p>The accuracy on the validation dataset, represented by <span class="math-container">$R$</span>, is actually non-differentiable, so the authors use the famous REINFORCE algorithm (which I will not explain here). The authors actually use an approximation of the REINFORCE algorithm (see section 3.2).</p>
  
  <p><span class="math-container">$$
  \frac{1}{m} \sum_{k=1}^m \sum_{t=1}^T \nabla_{\boldsymbol{\theta}_\text{c}} \log P (a_t \mid a_{{t-1}:1}; \boldsymbol{\theta}_\text{c}) R_k
  $$</span></p>
  
  <p>So, given this formulation and taking into account the formulation of the REINFORCE algorithm, then the state of the agent seems to be <span class="math-container">$a_{{t-1}:1}$</span>, that is, the previous actions of the agent. (Recall that the agent is a controller that is implemented as a <em>recurrent</em> neural network).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-19T10:24:44.290Z
Answers.Text: Thank you for the great answer! Just one follow up question:
  
  So calculation for the next hyperparameter would be done like $ \text{activation_function}(\boldsymbol{\theta} \times a_{t-1} + \text{bias}) $ where usually softmax is chosen I guess? And the last action at step 0 would be just a 0 vector right?
Answers.UserDisplayName: jaaq
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-19T17:38:33.267Z
Answers.Text: @jaaq I think that your reasoning makes sense. However, I would suggest you look at this implementation of ENAS (efficient NAS), [https://github.com/carpedm20/ENAS-pytorch](https://github.com/carpedm20/ENAS-pytorch), so that you have an idea of what people have done.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-18T23:25:15.663Z
Answers.LastActivityDate: 2019-07-19T17:24:58.317Z
Answers.LastEditDate: 2019-07-19T17:24:58.317Z
Answers.UserDisplayName: nbro
Body: <p>I have read through many of the papers and articles linked in <a href="https://ai.stackexchange.com/questions/12434/how-is-neural-architecture-search-performed">this thread</a> but I haven't been able to find an answer to my question.</p>
  
  <p>I have built some small RL networks and I understand how REINFORCE works. I don't quite understand how they are applied to NAS though. Usually RL agents map a <a href="https://en.wikipedia.org/wiki/File:Reinforcement_learning_diagram.svg" rel="nofollow noreferrer">state to an action</a> and get a reward so they can improve their decision making (which action to choose). I understand that the reward comes from the accuracy of the child network and the action is a series of digits encoding the network architecture. </p>
  
  <p>What is passed as the state to the RL agent? This doesn't seem to be mentioned in the papers and articles I read. Is it the previous network? Example input data?</p>
  
Tags: <reinforcement-learning><neural-architecture-search>
Title: How does RL based neural architecture search work?
UserDisplayName: jaaq
ThreadText: # How does RL based neural architecture search work?
  
  (Asked by: username_0 on 2019-07-18)
  
  I have read through many of the papers and articles linked in [this thread](https://ai.stackexchange.com/questions/12434/how-is-neural-architecture-search-performed) but I haven't been able to find an answer to my question.
  
  I have built some small RL networks and I understand how REINFORCE works. I don't quite understand how they are applied to NAS though. Usually RL agents map a [state to an action](https://en.wikipedia.org/wiki/File:Reinforcement_learning_diagram.svg) and get a reward so they can improve their decision making (which action to choose). I understand that the reward comes from the accuracy of the child network and the action is a series of digits encoding the network architecture.
  
  What is passed as the state to the RL agent? This doesn't seem to be mentioned in the papers and articles I read. Is it the previous network? Example input data?
  
  **Tags:** reinforcement-learning, neural-architecture-search
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-07-18. Score: 2 | Accepted answer
  
  (I will repeat a few details that you're already aware of, so that other users can also understand the context).
  
  In the [Neural Architecture Search (NAS) paper](https://arxiv.org/pdf/1611.01578.pdf) (that I mention [in my answer](https://ai.stackexchange.com/a/12435/2444) to the question you link to in your question), the agent is the controller (see also this question [Is there any difference between a control and an action in reinforcement learning?](https://ai.stackexchange.com/q/13119/2444)), which is implemented as a recurrent neural network (RNN). This controller produces actions (or controls), which are strings that represent the hyper-parameters of a neural network (see e.g. section 3.2), based on the reward that it receives, which, in this case, is the accuracy on the validation dataset that the designed (by the controller) and trained neural network obtains.
  
  In this context, the controller is thus the reinforcement learning agent (or policy). The controller is an RNN that is represented by a vector of parameters, $\boldsymbol{\theta}\_\text{c}$. This controller is trained using a policy gradient method to maximize the expected accuracy on the validation dataset (see e.g. section 3 of the paper). So $\boldsymbol{\theta}\_\text{c}$ are adjusted using this policy gradient method so that the controller generates NN architectures that, after trained on some task, produce higher accuracy on a validation dataset.
  
  The objective function of the controller is thus
  
  $$
  J(\boldsymbol{\theta}\_\text{c}) = \mathbb{E}[R]
  $$
  
  where $R$ is the accuracy on the validation dataset. Why *expected*? You will be performing this operation multiple times, so, intuitively, you want an average of the accuracy that you obtain on the validation dataset using multiple neural network architectures that the controller might produce.
  
  The accuracy on the validation dataset, represented by $R$, is actually non-differentiable, so the authors use the famous REINFORCE algorithm (which I will not explain here). The authors actually use an approximation of the REINFORCE algorithm (see section 3.2).
  
  $$
  \frac{1}{m} \sum\_{k=1}^m \sum\_{t=1}^T \nabla\_{\boldsymbol{\theta}\_\text{c}} \log P (a\_t \mid a\_{{t-1}:1}; \boldsymbol{\theta}\_\text{c}) R\_k
  $$
  
  So, given this formulation and taking into account the formulation of the REINFORCE algorithm, then the state of the agent seems to be $a\_{{t-1}:1}$, that is, the previous actions of the agent. (Recall that the agent is a controller that is implemented as a *recurrent* neural network).
  
  > **Comments:**
  > **username_0**: Thank you for the great answer! Just one follow up question:  So calculation for the next hyperparameter would be done like $ \text{activation_function}(\boldsymbol{\theta} \times a_{t-1} + \text{bias}) $ where usually softmax is chosen I guess? And the last action at step 0 would be just a 0 vector right?  
  > **username_1**: @jaaq I think that your reasoning makes sense. However, I would suggest you look at this implementation of ENAS (efficient NAS), [https://github.com/carpedm20/ENAS-pytorch](https://github.com/carpedm20/ENAS-pytorch), so that you have an idea of what people have done.  
  
  ---
  
  