Body: <p>I'm working on a project where I have access to position coordinates and velocity components of multiple agents in an environment. Assuming that one agent is controllable while others are not controllable, the velocity of the controllable agent represents the action in the defined MDP. can I consider the velocity of the controllable agent as part of the state space as well? i.e. for time step t the state includes the &quot;current&quot; velocity of the agent, and the action represents the velocity at the &quot;next time step&quot;. If this is not possible, then what are the consequences of doing so?
  I was looking for references that may give directions to answer the question, but couldn't find something useful, unfortunately.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-09-15T10:25:16.580Z
Comments.Text: There's no reason I can think about that would prevent you from doing this. A question though, what environment are you trying to model ? If you use velocity as an action, your agent will move in an unusual manner, being able to go very fast or to stop instantly, is that what you want ? Otherwise using acceleration as an action might be more desirable.
Comments.UserDisplayName: nathan raynal
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-09-15T12:24:39.670Z
Comments.Text: @nathanraynal Thanks Nathan for your comment! I'm trying to model self driving cars environment. I don't have access to "instantaneous" acceleration information. But, can I control the behavior of the agent via reward reshaping so as to avoid unusual behavior?
Comments.UserDisplayName: Eman.suradi
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-09-15T14:15:00.783Z
Comments.Text: You could define your set of available actions as a function of your state, if your state contains the velocity of the car. A simple choice would be for state $S_t = (..., ..., ..., v_t)$.
Comments.UserDisplayName: nathan raynal
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-09-15T14:27:44.933Z
Comments.Text: A simple choice would be for state $S_t = (..., ..., ..., v_t)$ to choose something like the set of action to be  $A_t = f(S_t) = [v_t - \alpha; v_t + \alpha]$, with $\alpha$ a small value. Maybe you can also shape your reward as to discourage velocity too far from the current one too.
Comments.UserDisplayName: nathan raynal
Tags: <reinforcement-learning><state-spaces><action-spaces>
Title: Can action space be part of the state space?
UserDisplayName: Eman.suradi
ThreadText: # Can action space be part of the state space?
  
  (Asked by: username_0 on 2022-09-15)
  
  I'm working on a project where I have access to position coordinates and velocity components of multiple agents in an environment. Assuming that one agent is controllable while others are not controllable, the velocity of the controllable agent represents the action in the defined MDP. can I consider the velocity of the controllable agent as part of the state space as well? i.e. for time step t the state includes the "current" velocity of the agent, and the action represents the velocity at the "next time step". If this is not possible, then what are the consequences of doing so?
  I was looking for references that may give directions to answer the question, but couldn't find something useful, unfortunately.
  
  **Tags:** reinforcement-learning, state-spaces, action-spaces
  
  > **Comments:**
  > **username_1**: There's no reason I can think about that would prevent you from doing this. A question though, what environment are you trying to model ? If you use velocity as an action, your agent will move in an unusual manner, being able to go very fast or to stop instantly, is that what you want ? Otherwise using acceleration as an action might be more desirable.  
  > **username_0**: @username_1 Thanks username_1 for your comment! I'm trying to model self driving cars environment. I don't have access to "instantaneous" acceleration information. But, can I control the behavior of the agent via reward reshaping so as to avoid unusual behavior?  
  > **username_1**: You could define your set of available actions as a function of your state, if your state contains the velocity of the car. A simple choice would be for state $S_t = (..., ..., ..., v_t)$.  
  > **username_1**: A simple choice would be for state $S_t = (..., ..., ..., v_t)$ to choose something like the set of action to be  $A_t = f(S_t) = [v_t - \alpha; v_t + \alpha]$, with $\alpha$ a small value. Maybe you can also shape your reward as to discourage velocity too far from the current one too.  
  
  ---
  
  ## 0 answers
  
  