Answers.Body: <p>If you build a function like <span class="math-container">$Q(s,a)$</span> using DQN, you have the problem that given 100 actions, you'll need 100 forward pass of your network</p>
  <p>Now, since neural networks can handle multiple outputs, we usually (not always, for example not for continuous action spaces) just  model <span class="math-container">$Q(s)$</span>, where the network outputs all the Qs for all the actions. This way, you only need 1 forward pass to get all the Qs.</p>
  <p>Also, bare in mind that using a 1-hot encoding, even though they make perfect sense for tabular cases, as state representation might not be the best for neural networks, as it will be much harder for them to generalize, and you'll have a giant input space</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-03T12:46:52.030Z
Answers.Text: OP's examples are actually a Q table implemented via a weights matrix, presumably as a teaching method, or to allow later upgrade of the implementation without re-writing from scratch.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-03T12:58:51.037Z
Answers.Text: @NeilSlater yes, i agree with you, but i don't get your point... do you think the answer is OT or what?...
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-03T14:03:19.627Z
Answers.Text: I was addressing your last para - you are critiquing approach explained in the question, but I think OP is coming from a very different starting place, and it might be an idea to unpick that to help them understand your advice.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-03T14:57:32.700Z
Answers.Text: @NeilSlater oh ok I see your point, I'll add a note on that, you're right
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-03T23:57:26.150Z
Answers.Text: Thank you for your answers! May I know what OP and OT stand for?
Answers.UserDisplayName: fermented_bean
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-04T00:11:33.153Z
Answers.Text: @fermented_bean OP=original poster, OT=off topic
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-04T00:33:09.773Z
Answers.Text: Oh I see, thought they were the abbreviation of the approaches. Btw, could I ask more about the case where the action is continuous? In such a case is learning q(s,a) the only choice? (a becomes a continuous feature or so)
Answers.UserDisplayName: fermented_bean
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-04T11:45:37.537Z
Answers.Text: @fermented_bean in that case you have a problem that even though defining $Q(s,a)$ might work, you  have to keep in mind that Qs are expected returns under a policy... but you don't have a policy... so you have to resort either to methods like DDPG/TD3/SAC, or to approaches that changes completely the paradigm like CMA-ES/CEM
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-03T11:59:26.817Z
Answers.LastActivityDate: 2023-12-03T14:58:17.257Z
Answers.LastEditDate: 2023-12-03T14:58:17.257Z
Answers.UserDisplayName: Alberto
Body: <p>Give the one-hot state vector <span class="math-container">$\boldsymbol{x}(s)=[x_1(s),x_2(s)]^T$</span> and action spaces <span class="math-container">$A(s)=\{a_1,a_2\}$</span> for all <span class="math-container">$s$</span>.</p>
  <p>In a course, I was taught to construct &quot;stack&quot; input vectors like <span class="math-container">$[x_{11}(s,a),x_{21}(s,a),x_{12}(s,a),x_{22}(s,a)]^T$</span> with <span class="math-container">$x_{ij}(s)=1$</span> if the action is <span class="math-container">$a_j$</span> and the state is <span class="math-container">$s_i$</span>, and the output is just <span class="math-container">$q(s,a)$</span>.</p>
  <p>However, in DQN, the input vector is <span class="math-container">$\boldsymbol{x}(s)=[x_1(s),x_2(s)]^T$</span>, and the output vector is <span class="math-container">$[q(s,a_1),q(s,a_2)]$</span>.</p>
  <p>My question is: What are the usual ways to construct input and ouput vectors in  Reinforcement Learning to learn the action value function q(s,a)? Are there any other methods?</p>
  
Tags: <reinforcement-learning><feature-selection>
Title: Methods of constructing input and ouput vectors in Reinforcement Learning with approximation function learning?
UserDisplayName: fermented_bean
ThreadText: # Methods of constructing input and ouput vectors in Reinforcement Learning with approximation function learning?
  
  (Asked by: username_0 on 2023-12-03)
  
  Give the one-hot state vector $\boldsymbol{x}(s)=[x\_1(s),x\_2(s)]^T$ and action spaces $A(s)=\{a\_1,a\_2\}$ for all $s$.
  
  In a course, I was taught to construct "stack" input vectors like $[x\_{11}(s,a),x\_{21}(s,a),x\_{12}(s,a),x\_{22}(s,a)]^T$ with $x\_{ij}(s)=1$ if the action is $a\_j$ and the state is $s\_i$, and the output is just $q(s,a)$.
  
  However, in DQN, the input vector is $\boldsymbol{x}(s)=[x\_1(s),x\_2(s)]^T$, and the output vector is $[q(s,a\_1),q(s,a\_2)]$.
  
  My question is: What are the usual ways to construct input and ouput vectors in Reinforcement Learning to learn the action value function q(s,a)? Are there any other methods?
  
  **Tags:** reinforcement-learning, feature-selection
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-12-03. Score: 1 | Accepted answer
  
  If you build a function like $Q(s,a)$ using DQN, you have the problem that given 100 actions, you'll need 100 forward pass of your network
  
  Now, since neural networks can handle multiple outputs, we usually (not always, for example not for continuous action spaces) just model $Q(s)$, where the network outputs all the Qs for all the actions. This way, you only need 1 forward pass to get all the Qs.
  
  Also, bare in mind that using a 1-hot encoding, even though they make perfect sense for tabular cases, as state representation might not be the best for neural networks, as it will be much harder for them to generalize, and you'll have a giant input space
  
  > **Comments:**
  > **username_2**: OP's examples are actually a Q table implemented via a weights matrix, presumably as a teaching method, or to allow later upgrade of the implementation without re-writing from scratch.  
  > **username_1**: @username_2 yes, i agree with you, but i don't get your point... do you think the answer is OT or what?...  
  > **username_2**: I was addressing your last para - you are critiquing approach explained in the question, but I think OP is coming from a very different starting place, and it might be an idea to unpick that to help them understand your advice.  
  > **username_1**: @username_2 oh ok I see your point, I'll add a note on that, you're right  
  > **username_0**: Thank you for your answers! May I know what OP and OT stand for?  
  > **username_1**: @username_0 OP=original poster, OT=off topic  
  > **username_0**: Oh I see, thought they were the abbreviation of the approaches. Btw, could I ask more about the case where the action is continuous? In such a case is learning q(s,a) the only choice? (a becomes a continuous feature or so)  
  > **username_1**: @username_0 in that case you have a problem that even though defining $Q(s,a)$ might work, you  have to keep in mind that Qs are expected returns under a policy... but you don't have a policy... so you have to resort either to methods like DDPG/TD3/SAC, or to approaches that changes completely the paradigm like CMA-ES/CEM  
  
  ---
  
  