Answers.Body: <p>There are more possible approaches to tackle this problem:</p>
  
  <ol>
  <li><p>Use a reinforcement learning method which can cope with a continous state space. This would eliminate the need for discretizing the state space which in turn, if I understood correctly, leads to problems in transitioning between states. You can also consider selecting a reinforcement learning method that can cope with a continous state space and a continous action space. </p></li>
  <li><p>Actions do not have to be immediate. If your action 1 is transitioning form <span class="math-container">$S_i$</span> to <span class="math-container">$S_{i+1}$</span> you can have a function wich continously increases the spead until the next state is reached and only then you consider the execution of the action complete. Furthermore, the magnitude of the actions can be dependent on the current state, if it assures state transition. </p></li>
  <li><p>You can add more actions. Use RL for your advantage and define not just <span class="math-container">$+1$</span> and <span class="math-container">$-1$</span> but <span class="math-container">$+0.1$</span>, <span class="math-container">$0.3$</span>, <span class="math-container">$+0.5$</span>, <span class="math-container">$+0.7$</span>, <span class="math-container">$+1$</span> (and same for deacreasing the speed) etc. Add a negative reward for all actions which do not cause a state transition or which cause higher jump then needed. However, care must be taken to make sure you have the same speed fro each sampling. Eg. if you are in <span class="math-container">$S_1$</span> and the action +0.1 does not cause a state transition to <span class="math-container">$S_2$</span> you have to reset the state (if possible) or fail the epoch, since the state you are in is not only dependent on the action and the previous state. In other words the speed of e.g. <span class="math-container">$1 rad/s$</span> (with a tolarance band) defined as <span class="math-container">$S_1$</span> will be slightly higher and ending up in <span class="math-container">$S_2$</span> after applying a slight increase in the speed takes less change in the speed than from <span class="math-container">$S_1$</span> without the "unnoticed" speed change.</p></li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-12-14T21:23:41.230Z
Answers.Text: Ok thank you very much for a very nice explanation. Actually in my problem, I have finite states available (say 10). There are total 20 actions (say discrete value from 3 to 23). I know state 1 is bad and state 10 is good. In each state, i can only try few actions (not all 20). The problem actually lies here. When I do simulation, I know if I increase action by some number relative to previous action I will transition to next state. But In real implementation, it does not. Can I do like binary decisions (1 means keep increasing action and -1 keep decreasing until I transition to next state.
Answers.UserDisplayName: adnan jaffar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-12-14T21:26:03.323Z
Answers.Text: So you are training in a virtual environment (simulation) which does not reflect your real setup used in the expoitation?
Answers.UserDisplayName: 50k4
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-12-14T21:29:15.187Z
Answers.Text: Yes, I still did not implement it in real but i am worried before implementation. My real environment is patient and so I can only try when I am confident about my algorithm. The virtual environment is somehow relating with real but not 100 percent.
Answers.UserDisplayName: adnan jaffar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-12-14T21:36:04.097Z
Answers.Text: Can I make Q table like states * decision actions and decision actions are +1,-1. 1 means increase the previous action by some number and -1 decrease it by some number until i reach to next state. and I will update Q table only when  i transition to next state.
Answers.UserDisplayName: adnan jaffar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-12-14T21:43:03.017Z
Answers.Text: As stated in point 2, the action does not have to be immediate and if it brings your environment to the same state in reality as in simulation it does not have to be the same implementation or it does not have to act in the same way.
Answers.UserDisplayName: 50k4
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-12-14T21:45:15.250Z
Answers.Text: You can also have a different parameter in the action execution which links action magnitude to the current state. e.g for a slow speed state apply less magnitude. This does not necessarily have to be in the Q table.
Answers.UserDisplayName: 50k4
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-12-14T21:53:58.147Z
Answers.Text: Thank you for the perfect explanation. I got your point.
Answers.UserDisplayName: adnan jaffar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-12-14T13:13:42.293Z
Answers.LastActivityDate: 2018-12-14T22:03:46.020Z
Answers.LastEditDate: 2018-12-14T22:03:46.020Z
Answers.UserDisplayName: 50k4
Body: <p>How to select an action in a state if the action does not necesarily cause the environment to change state? </p>
  
  <p>Given 10 states (<span class="math-container">$S_0$</span> to <span class="math-container">$S_9$</span>) and in each state <span class="math-container">$i$</span> there are two actions defined <span class="math-container">$(1,-1)$</span>. <span class="math-container">$1$</span> increases a parameter of the environment and <span class="math-container">$-1$</span> decreases it.</p>
  
  <p>For example, if the parameter is speed and it is currently 1 rad/s, corresponding to <span class="math-container">${S_i}$</span>, an action can be either increase or decrease this and therefore transition to the next state, ideally <span class="math-container">$S_{i-1}$</span> or <span class="math-container">$S_{i+1}$</span>.</p>
  
  <p>It is unclear how to formulate a reinforcement learning problem in this case. The problem is the same magnitude of the parameter change through an action. </p>
  
  <p>The range of the parameter is (3, 25) and step size is 1. The problem is thath the responce of the environment is not the same in every satate. In some states a parameter change with magnitude 1 results in a state transition in some state the magintude proves to be to small to provoke a transition change. </p>
  
  <p>For example, if the envitonment is instate <span class="math-container">$S_1$</span> and action 1 is applied, how can the magintude of the paremter be adopted in a way which assures that the environment will transition to state <span class="math-container">$S_2$</span>?  How can the step in the paremeter change be made adaptive? Actually my environment is uncertain that is why I don't know exactly whether this action will take me to next state or not.</p>
  
  <p>For your information, I am using Q learning off policy algorithm. Suppose state 9 is the goal state and  my Q table is <span class="math-container">$10 \times 2$</span>.</p>
  
Tags: <reinforcement-learning>
Title: Reinforcement Learning with Adaptive Action Magnitude
UserDisplayName: adnan jaffar
ThreadText: # Reinforcement Learning with Adaptive Action Magnitude
  
  (Asked by: username_0 on 2018-12-13)
  
  How to select an action in a state if the action does not necesarily cause the environment to change state?
  
  Given 10 states ($S\_0$ to $S\_9$) and in each state $i$ there are two actions defined $(1,-1)$. $1$ increases a parameter of the environment and $-1$ decreases it.
  
  For example, if the parameter is speed and it is currently 1 rad/s, corresponding to ${S\_i}$, an action can be either increase or decrease this and therefore transition to the next state, ideally $S\_{i-1}$ or $S\_{i+1}$.
  
  It is unclear how to formulate a reinforcement learning problem in this case. The problem is the same magnitude of the parameter change through an action.
  
  The range of the parameter is (3, 25) and step size is 1. The problem is thath the responce of the environment is not the same in every satate. In some states a parameter change with magnitude 1 results in a state transition in some state the magintude proves to be to small to provoke a transition change.
  
  For example, if the envitonment is instate $S\_1$ and action 1 is applied, how can the magintude of the paremter be adopted in a way which assures that the environment will transition to state $S\_2$? How can the step in the paremeter change be made adaptive? Actually my environment is uncertain that is why I don't know exactly whether this action will take me to next state or not.
  
  For your information, I am using Q learning off policy algorithm. Suppose state 9 is the goal state and my Q table is $10 \times 2$.
  
  **Tags:** reinforcement-learning
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-12-14. Score: 0 | Accepted answer
  
  There are more possible approaches to tackle this problem:
  
  1. Use a reinforcement learning method which can cope with a continous state space. This would eliminate the need for discretizing the state space which in turn, if I understood correctly, leads to problems in transitioning between states. You can also consider selecting a reinforcement learning method that can cope with a continous state space and a continous action space.
  2. Actions do not have to be immediate. If your action 1 is transitioning form $S\_i$ to $S\_{i+1}$ you can have a function wich continously increases the spead until the next state is reached and only then you consider the execution of the action complete. Furthermore, the magnitude of the actions can be dependent on the current state, if it assures state transition.
  3. You can add more actions. Use RL for your advantage and define not just $+1$ and $-1$ but $+0.1$, $0.3$, $+0.5$, $+0.7$, $+1$ (and same for deacreasing the speed) etc. Add a negative reward for all actions which do not cause a state transition or which cause higher jump then needed. However, care must be taken to make sure you have the same speed fro each sampling. Eg. if you are in $S\_1$ and the action +0.1 does not cause a state transition to $S\_2$ you have to reset the state (if possible) or fail the epoch, since the state you are in is not only dependent on the action and the previous state. In other words the speed of e.g. $1 rad/s$ (with a tolarance band) defined as $S\_1$ will be slightly higher and ending up in $S\_2$ after applying a slight increase in the speed takes less change in the speed than from $S\_1$ without the "unnoticed" speed change.
  
  > **Comments:**
  > **username_0**: Ok thank you very much for a very nice explanation. Actually in my problem, I have finite states available (say 10). There are total 20 actions (say discrete value from 3 to 23). I know state 1 is bad and state 10 is good. In each state, i can only try few actions (not all 20). The problem actually lies here. When I do simulation, I know if I increase action by some number relative to previous action I will transition to next state. But In real implementation, it does not. Can I do like binary decisions (1 means keep increasing action and -1 keep decreasing until I transition to next state.  
  > **username_1**: So you are training in a virtual environment (simulation) which does not reflect your real setup used in the expoitation?  
  > **username_0**: Yes, I still did not implement it in real but i am worried before implementation. My real environment is patient and so I can only try when I am confident about my algorithm. The virtual environment is somehow relating with real but not 100 percent.  
  > **username_0**: Can I make Q table like states * decision actions and decision actions are +1,-1. 1 means increase the previous action by some number and -1 decrease it by some number until i reach to next state. and I will update Q table only when  i transition to next state.  
  > **username_1**: As stated in point 2, the action does not have to be immediate and if it brings your environment to the same state in reality as in simulation it does not have to be the same implementation or it does not have to act in the same way.  
  > **username_1**: You can also have a different parameter in the action execution which links action magnitude to the current state. e.g for a slow speed state apply less magnitude. This does not necessarily have to be in the Q table.  
  > **username_0**: Thank you for the perfect explanation. I got your point.  
  
  ---
  
  