Answers.Body: <blockquote>
    <p>So does that mean, that the input of the first hidden layer was simply the state and the input of the second hidden layer the output of the first hidden layer concatenated with the actions?</p>
  </blockquote>
  
  <p>Yes.</p>
  
  <blockquote>
    <p>Why would you do that? To have the first layer focus on learning the state value independent of the selected action? How would that help?</p>
  </blockquote>
  
  <p>Neural networks hidden layers learn representations that get progressively closer to a linear relationship with the target, layer by layer.</p>
  
  <p>So the first layer would not be learning the state value per se, but some representation of the state that was better related to the action value at the output.</p>
  
  <p>Neural network architectures are often established by experimentation, so I expect here they tried the idea and the performance was OK. If they do not give alternative architectures in the paper, then the precise reason is not clear.</p>
  
  <p>I can try a few guesses:</p>
  
  <ul>
  <li><p>Concatenating the actions with the states in the input layer would result in more parameters for the neural network to achieve the same accuracy, running slightly slower. That's because more weights would be required to link the input layer with the first hidden layer.</p></li>
  <li><p>Separating the layers of state and action inputs is a form of regularisation, as the first layer has to produce features that are useful for all possible actions.</p></li>
  <li><p>You don't want the neural network to construct a reverse map <span class="math-container">$\pi(a|s) \rightarrow Q(s,a)$</span>, you want it to independently assess the action values to do its job as a critic. By having the states and actions presented in different laters, this reduces the chance of the neural network finding shortcuts due to "recognising the policy" to predict the values. That's because the state and action pairings may repeat (so be recognised), but the first layer activations change over time (so even with repeated state/action pairs, this is a new representation to learn from).</p></li>
  </ul>
  
  <blockquote>
    <p>Is this just a small little tweak or a more significant improvement?</p>
  </blockquote>
  
  <p>I don't know, and suggest looking through previous work by the same authors in case they describe the approach in more detail.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-22T13:42:13.300Z
Answers.LastActivityDate: 2019-10-22T13:42:13.300Z
Answers.UserDisplayName: Neil Slater
Body: <p>I read the <a href="https://arxiv.org/abs/1509.02971" rel="nofollow noreferrer">DDPG paper</a>, in which the authors state that the actions are fed only later to their Q network:</p>
  
  <blockquote>
    <p>Actions were not included until the 2nd hidden layer of Q. (Sec 7, Experiment Details)</p>
  </blockquote>
  
  <p>So does that mean, that the input of the first hidden layer was simply the state and the input of the second hidden layer the output of the first hidden layer concatenated with the actions?</p>
  
  <p>Why would you do that? To have the first layer focus on learning the state value independent of the selected action? How would that help?</p>
  
  <p>Is this just a small little tweak or a more significant improvement?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-12-26T14:17:03.040Z
Comments.Text: > Is this just a small little tweak or a more significant improvement? This seems pretty major.. I tested this with and without this tweak and without it the networks just wouldn't learn at all. But it may depend on the environment too, I'm not sure.
Comments.UserDisplayName: tache
Tags: <neural-networks><reinforcement-learning><q-learning><markov-decision-process><ddpg>
Title: Why feed actions in later layer in Q network?
UserDisplayName: stefanbschneider
ThreadText: # Why feed actions in later layer in Q network?
  
  (Asked by: username_0 on 2019-10-22)
  
  I read the [DDPG paper](https://arxiv.org/abs/1509.02971), in which the authors state that the actions are fed only later to their Q network:
  
  > Actions were not included until the 2nd hidden layer of Q. (Sec 7, Experiment Details)
  
  So does that mean, that the input of the first hidden layer was simply the state and the input of the second hidden layer the output of the first hidden layer concatenated with the actions?
  
  Why would you do that? To have the first layer focus on learning the state value independent of the selected action? How would that help?
  
  Is this just a small little tweak or a more significant improvement?
  
  **Tags:** neural-networks, reinforcement-learning, q-learning, markov-decision-process, ddpg
  
  > **Comments:**
  > **username_1**: > Is this just a small little tweak or a more significant improvement? This seems pretty major.. I tested this with and without this tweak and without it the networks just wouldn't learn at all. But it may depend on the environment too, I'm not sure.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2019-10-22. Score: 2 | Accepted answer
  
  > So does that mean, that the input of the first hidden layer was simply the state and the input of the second hidden layer the output of the first hidden layer concatenated with the actions?
  
  Yes.
  
  > Why would you do that? To have the first layer focus on learning the state value independent of the selected action? How would that help?
  
  Neural networks hidden layers learn representations that get progressively closer to a linear relationship with the target, layer by layer.
  
  So the first layer would not be learning the state value per se, but some representation of the state that was better related to the action value at the output.
  
  Neural network architectures are often established by experimentation, so I expect here they tried the idea and the performance was OK. If they do not give alternative architectures in the paper, then the precise reason is not clear.
  
  I can try a few guesses:
  
  * Concatenating the actions with the states in the input layer would result in more parameters for the neural network to achieve the same accuracy, running slightly slower. That's because more weights would be required to link the input layer with the first hidden layer.
  * Separating the layers of state and action inputs is a form of regularisation, as the first layer has to produce features that are useful for all possible actions.
  * You don't want the neural network to construct a reverse map $\pi(a|s) \rightarrow Q(s,a)$, you want it to independently assess the action values to do its job as a critic. By having the states and actions presented in different laters, this reduces the chance of the neural network finding shortcuts due to "recognising the policy" to predict the values. That's because the state and action pairings may repeat (so be recognised), but the first layer activations change over time (so even with repeated state/action pairs, this is a new representation to learn from).
  
  > Is this just a small little tweak or a more significant improvement?
  
  I don't know, and suggest looking through previous work by the same authors in case they describe the approach in more detail.
  
  
  ---
  
  