Answers.Body: <p>Ultimately, a policy must be such that is is possible for an agent to execute it.</p>
  <p>If the policy depends on the state, the implicit assumption is that the agent has knowledge of the state and can therefore choose its actions accordingly. This is the common case of a MDP as an underlying framework for RL.</p>
  <p>If the state is not known to the agent, it may instead perceive observations  that have some relation to the state (although they might not fully reveal the state). Then, one can condition policies on the last received observation.</p>
  <p>However, it is useful to note that conditioning on the last observation in a partially observable setting is in general not sufficient for acting optimally. There are cases where one may need to remember a longer history of observations to decide which action is best. In general, acting optimally in such a partially observable setting requires that the policy is a function of the <em>complete history</em> of past actions taken and observations perceived. The underlying framework is then the <em>partially observable</em> MDP, or POMDP.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-22T12:50:36.160Z
Answers.Text: This is a good answer, but one thing is missing. For example, in DQN applied to Atari games, are you going to learn a policy from states to actions or from observations to actions? In DQN, you're basically applying Q-learning with function approximation to POMDPs, which you assume to be MDPs by approximating states as a sequence of, say, 4 observations. So, in this case, will you use this stack of observations as an approximation to the current state even after having stopped learning, so will the policy, effectively, be a mapping from a sequence of successive observations to actions?
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-22T14:59:18.697Z
Answers.Text: @nbro In an MDP, the current state and action are sufficient to predict the next state, and knowing any older past states will not help to predict better. The "trick" in DQN uses a stack of frames and calls it "the state" and then the problem is solved like it was an MDP. But the stack of frames is not a state in the MDP sense above: it's not the only thing besides the action that matters for the next observation stack. In an Atari game, an MDP state could be the state of the internal game memory. That and the current input determine the next memory state. Trick in DQN is an approximation.
Answers.UserDisplayName: mikkola
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-21T15:06:34.343Z
Answers.LastActivityDate: 2021-09-21T15:06:34.343Z
Answers.UserDisplayName: mikkola
Body: <p>I am new to RL and I am following Sutton &amp; Barto's book.</p>
  <p>My doubt is, when we talk about the policy of our agent, we say it is the probability of taking some action <span class="math-container">$a$</span> given the <em>state</em> <span class="math-container">$s$</span>. However, I think that the policy should be defined in terms of <em>observations</em> and not states because I think it is not always possible for an agent to fully capture the state due to various reasons, maybe lack of sensors or lack of memory.</p>
  <p>So, why are policies defined as functions of states and not observations?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-19T04:47:12.603Z
Comments.Text: From what I remember, RL can handle POMDP.
Comments.UserDisplayName: FourierFlux
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-19T05:10:48.367Z
Comments.Text: oh, I'll read about it, because I think one of  the reason  they defined it that way can be  that, they mentioned the environment to be fully observable. But then how my agent is going to make actions? if i dont know what state I am in. It is all very  confusing in my head @FourierFlux
Comments.UserDisplayName: hakiki_makato
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-19T05:19:23.350Z
Comments.Text: From what I remember something in Sutton's book says function based approximations can work on POMDP, but discrete state cannot. I would need to look it up though to be sure.
Comments.UserDisplayName: FourierFlux
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-19T05:22:35.767Z
Comments.Text: To be honest though it shouldn't work intuitively speaking, since an RL problem should ultimately be a MDP and there cannot be hidden states which have memory so I do not know.
Comments.UserDisplayName: FourierFlux
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-19T08:58:24.060Z
Comments.Text: @FourierFlux: I think you have remembered correctly. The key is that there are different degrees and types of missing knowledge that can impact an agent learning to act in an MDP. I think a good answer will try to cover that, as well as how observatons and state can interact.
Comments.UserDisplayName: Neil Slater
Tags: <reinforcement-learning><definitions><markov-decision-process><policies>
Title: In reinforcement learning, why are policies defined as functions of states and not observations?
UserDisplayName: hakiki_makato
ThreadText: # In reinforcement learning, why are policies defined as functions of states and not observations?
  
  (Asked by: username_0 on 2021-09-19)
  
  I am new to RL and I am following Sutton & Barto's book.
  
  My doubt is, when we talk about the policy of our agent, we say it is the probability of taking some action $a$ given the *state* $s$. However, I think that the policy should be defined in terms of *observations* and not states because I think it is not always possible for an agent to fully capture the state due to various reasons, maybe lack of sensors or lack of memory.
  
  So, why are policies defined as functions of states and not observations?
  
  **Tags:** reinforcement-learning, definitions, markov-decision-process, policies
  > **Comments:**
  > **username_1**: From what I remember, RL can handle POMDP.  
  > **username_0**: oh, I'll read about it, because I think one of  the reason  they defined it that way can be  that, they mentioned the environment to be fully observable. But then how my agent is going to make actions? if i dont know what state I am in. It is all very  confusing in my head @FourierFlux  
  > **username_1**: From what I remember something in Sutton's book says function based approximations can work on POMDP, but discrete state cannot. I would need to look it up though to be sure.  
  > **username_1**: To be honest though it shouldn't work intuitively speaking, since an RL problem should ultimately be a MDP and there cannot be hidden states which have memory so I do not know.  
  > **username_2**: @FourierFlux: I think you have remembered correctly. The key is that there are different degrees and types of missing knowledge that can impact an agent learning to act in an MDP. I think a good answer will try to cover that, as well as how observatons and state can interact.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2021-09-21. Score: 4 | Accepted answer
  
  Ultimately, a policy must be such that is is possible for an agent to execute it.
  
  If the policy depends on the state, the implicit assumption is that the agent has knowledge of the state and can therefore choose its actions accordingly. This is the common case of a MDP as an underlying framework for RL.
  
  If the state is not known to the agent, it may instead perceive observations that have some relation to the state (although they might not fully reveal the state). Then, one can condition policies on the last received observation.
  
  However, it is useful to note that conditioning on the last observation in a partially observable setting is in general not sufficient for acting optimally. There are cases where one may need to remember a longer history of observations to decide which action is best. In general, acting optimally in such a partially observable setting requires that the policy is a function of the *complete history* of past actions taken and observations perceived. The underlying framework is then the *partially observable* MDP, or POMDP.
  
  > **Comments:**
  > **username_4**: This is a good answer, but one thing is missing. For example, in DQN applied to Atari games, are you going to learn a policy from states to actions or from observations to actions? In DQN, you're basically applying Q-learning with function approximation to POMDPs, which you assume to be MDPs by approximating states as a sequence of, say, 4 observations. So, in this case, will you use this stack of observations as an approximation to the current state even after having stopped learning, so will the policy, effectively, be a mapping from a sequence of successive observations to actions?  
  > **username_3**: @nbro In an MDP, the current state and action are sufficient to predict the next state, and knowing any older past states will not help to predict better. The "trick" in DQN uses a stack of frames and calls it "the state" and then the problem is solved like it was an MDP. But the stack of frames is not a state in the MDP sense above: it's not the only thing besides the action that matters for the next observation stack. In an Atari game, an MDP state could be the state of the internal game memory. That and the current input determine the next memory state. Trick in DQN is an approximation.  
  
  ---
  
  