Answers.Body: <blockquote>
    <p>In the above situation I mentioned, wont the agent think due to 'move right' action the state has changed, which is not really the case?</p>
  </blockquote>
  
  <p>Generally the agent does not need to "think" what the next state will be. After taking an action, it should <em>observe</em> the reward and the next state. </p>
  
  <p>The logic for the next state is controlled by the environment. In your case, the environment of the simulator should accept the action choice and then not allow the sideways movement off the road, and present the correct logical next state. It may in addition alter the reward signal if you want to penalise this choice (in comments you say this is not done - that's fine, it's a choice you make on problem set up, what behaviour to reward or penalise).</p>
  
  <p>A more advanced agent might also try to <em>predict</em> what the next state will be, especially if it was using a planning algorithm (e.g. Dyna-Q). Even then there is no reason or rule that it should have to predict an impossible next state - a well-trained agent should in fact predict correctly that moving right is same as "do nothing" when the car is in far right-hand lane. For a value-based approach, like Q-learning the agent may do this simply by scoring the two actions with the same value.</p>
  
  <p>In your case, you do not need to try and second-guess what the agent will predict. Provided your state information is enough for the agent to observe that it is in the far left or far right lane, it should [eventually, after training] correctly predict the value functions and/or optimal actions as it learns the environment. That includes, after a few examples, that it will predict the outcome when moving right when already on right hand side. For that to happen reliably, you do need to check your state representation is adequate, and function approximation hyper-parameters are suited to the problem (the latter part could take some experimentation)</p>
  
  <p>If the state data that the car receives does not contain enough information for the agent to tell that it is in extreme left or right lane, then you will have a problem. This is entirely possible in general, although from the image you show, that does not seem to be an issue. </p>
  
  <p>One possible related issue is if the other cars in the scene all have different velocities. This velocity information is not shown in the state, and could cause your agent problems as it cannot tell the difference between a stationary car in front of it, and one which better matches its own velocity. You could treat this missing data as a partially observable state, and adjust you agent. Or you might be able to do the same as the Deep Mind DQN team for Atari games and use a sequence of e.g. last 4 images as the state, which should then include enough velocity data to make reliable estimates of state/action values.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-01-18T15:45:13.440Z
Answers.LastActivityDate: 2018-01-18T16:09:13.940Z
Answers.LastEditDate: 2018-01-18T16:09:13.940Z
Answers.UserDisplayName: Neil Slater
Body: <p><a href="https://i.sstatic.net/Nm8RW.jpgT" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Nm8RW.jpgT" alt="enter image description here"></a></p>
  
  <p>The above environment is <a href="https://selfdrivingcars.mit.edu/deeptraffic/" rel="nofollow noreferrer">DeepTraffic</a> </p>
  
  <p>Now consider this situation in the above environment, the Red car (we control it with our RL agent) is on the extreme right lane.</p>
  
  <p>During the exploration phase, we take a 'move right' action, which ofcourse will result in the car not moving right,but the other cars will be moving, state changes due to the rules of the environment.</p>
  
  <p>I'm using CNN to solve this, the state representation is the image itself and its a Q-Learning algorithm as described in DQN paper from deepmind.</p>
  
  <p>In the above situation I mentioned, wont the agent think due to 'move right' action the state has changed, which is not really the case?</p>
  
  <p>and when remembering the state transition (s,a,r,s') should i remember the actual action 'move right'(invalid) or 'do nothing'(correct as per env) ?</p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2018-01-18T12:53:12.253Z
Comments.Text: If the agent goes right as opposed to donny nothing, does it get a different reward?
Comments.UserDisplayName: Jaden Travnik
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2018-01-18T13:44:29.627Z
Comments.Text: 'Do Nothing' action has 0 reward on all states. In this situation go-right will also have a 0 reward, since no damage is done.
Comments.UserDisplayName: Naveen
Tags: <reinforcement-learning>
Title: RL agent's view of state transitions
UserDisplayName: Naveen
ThreadText: # RL agent's view of state transitions
  
  (Asked by: username_0 on 2018-01-18)
  
  [![enter image description here](https://i.sstatic.net/Nm8RW.jpgT)](https://i.sstatic.net/Nm8RW.jpgT)
  
  The above environment is [DeepTraffic](https://selfdrivingcars.mit.edu/deeptraffic/)
  
  Now consider this situation in the above environment, the Red car (we control it with our RL agent) is on the extreme right lane.
  
  During the exploration phase, we take a 'move right' action, which ofcourse will result in the car not moving right,but the other cars will be moving, state changes due to the rules of the environment.
  
  I'm using CNN to solve this, the state representation is the image itself and its a Q-Learning algorithm as described in DQN paper from deepmind.
  
  In the above situation I mentioned, wont the agent think due to 'move right' action the state has changed, which is not really the case?
  
  and when remembering the state transition (s,a,r,s') should i remember the actual action 'move right'(invalid) or 'do nothing'(correct as per env) ?
  
  **Tags:** reinforcement-learning
  > **Comments:**
  > **username_1**: If the agent goes right as opposed to donny nothing, does it get a different reward?  
  > **username_0**: 'Do Nothing' action has 0 reward on all states. In this situation go-right will also have a 0 reward, since no damage is done.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2018-01-18. Score: 1 | Accepted answer
  
  > In the above situation I mentioned, wont the agent think due to 'move right' action the state has changed, which is not really the case?
  
  Generally the agent does not need to "think" what the next state will be. After taking an action, it should *observe* the reward and the next state.
  
  The logic for the next state is controlled by the environment. In your case, the environment of the simulator should accept the action choice and then not allow the sideways movement off the road, and present the correct logical next state. It may in addition alter the reward signal if you want to penalise this choice (in comments you say this is not done - that's fine, it's a choice you make on problem set up, what behaviour to reward or penalise).
  
  A more advanced agent might also try to *predict* what the next state will be, especially if it was using a planning algorithm (e.g. Dyna-Q). Even then there is no reason or rule that it should have to predict an impossible next state - a well-trained agent should in fact predict correctly that moving right is same as "do nothing" when the car is in far right-hand lane. For a value-based approach, like Q-learning the agent may do this simply by scoring the two actions with the same value.
  
  In your case, you do not need to try and second-guess what the agent will predict. Provided your state information is enough for the agent to observe that it is in the far left or far right lane, it should [eventually, after training] correctly predict the value functions and/or optimal actions as it learns the environment. That includes, after a few examples, that it will predict the outcome when moving right when already on right hand side. For that to happen reliably, you do need to check your state representation is adequate, and function approximation hyper-parameters are suited to the problem (the latter part could take some experimentation)
  
  If the state data that the car receives does not contain enough information for the agent to tell that it is in extreme left or right lane, then you will have a problem. This is entirely possible in general, although from the image you show, that does not seem to be an issue.
  
  One possible related issue is if the other cars in the scene all have different velocities. This velocity information is not shown in the state, and could cause your agent problems as it cannot tell the difference between a stationary car in front of it, and one which better matches its own velocity. You could treat this missing data as a partially observable state, and adjust you agent. Or you might be able to do the same as the Deep Mind DQN team for Atari games and use a sequence of e.g. last 4 images as the state, which should then include enough velocity data to make reliable estimates of state/action values.
  
  
  ---
  
  