Answers.Body: <p>An environment is partially observable if the agent cannot <em>fully</em> observe the current state but it only <em>partially</em> observes it. More specifically, in <em>fully</em> observable MDPs (FOMDPs), the agent knows the current state of the environment, which <em>can or not</em> (for example, depending on whether the state is Markov or not) contain all the information required to theoretically take the optimal action. In a <em>partially</em> observable MDPs (POMDPs), the state is not available or observable, so the agent might not possess all the required information to theoretically take the optimal action. </p>
  
  <p>For example, in the game of poker, there are hidden and observable cards. In this game, the state of the environment <em>could</em> be defined as all the hands of all players, the cards on the table and the next cards that will be drawn from the deck until the end of the round. If a player had access to all this information, then the environment would be fully observable, but, usually, this information is not available, so poker is usually considered a partially observable game. </p>
  
  <p>You <em>defined</em> the state as follows</p>
  
  <p><span class="math-container">$$s_t = \{ p_{t:t-n}, E_t\}$$</span></p>
  
  <p>So, if you have access to the past <span class="math-container">$n$</span> price records, <span class="math-container">$p_{1:n}$</span>, and the energy level, <span class="math-container">$E_t$</span>, then your environment is fully observable. However, this does not mean that it is a Markov environment, that is, intuitively, that your current state is sufficient to theoretically take the optimal action. </p>
  
  <p>To conclude, partial observability depends on the definition of state in a specific problem. The concept of Markov property is also related to the concept of partial observability. However, in the case of MDPs, the concept of Markov property has a specific definition: given the current state and an action, the probability of the next state is conditionally independent of all previous states and actions or, more formally</p>
  
  <p><span class="math-container">$$
  P(S_{t+1} \mid S_t, A_t) = P(S_{t+1} \mid S_t, S_{t-1}, \dots, S_1, A_t, A_{t-1}, \dots, A_1)
  $$</span></p>
  
  <p>where <span class="math-container">$S_{t+1}$</span> is the next state, <span class="math-container">$S_t$</span> the current state, <span class="math-container">$A_t$</span> the current action, <span class="math-container">$S_{t-1}, \dots, S_1$</span> the previous states and <span class="math-container">$A_{t-1}, \dots, A_1$</span> the previous actions.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-01T05:32:44.750Z
Answers.Text: Thank you for clear explanationI have an additional question. As you said in your answer, to find optimal action it will be needed to give sufficient state information (Markov environment). If that is true, my state representation in the problem explained in my question is not providing sufficient information because it provides 'past n-step price' rather than 'the price on the hour it trades'. I intended to give helpful information to infer the price on the hour because practically the price on that time is not known.
Answers.UserDisplayName: JH Lee
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-01T05:32:49.740Z
Answers.Text: In this case, am I wrongly defining the problem or is it something that reinforcement learning cannot solve?
Answers.UserDisplayName: JH Lee
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-01T10:48:23.903Z
Answers.Text: @JHLee I would ask this as a new question on the website. I would also add details related to your definition of the reward function and possible actions.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-01T01:24:19.210Z
Answers.LastActivityDate: 2019-09-01T10:41:17.093Z
Answers.LastEditDate: 2019-09-01T10:41:17.093Z
Answers.UserDisplayName: nbro
Body: <p>I'm currently working on a project to make an DQN agent that decides whether to charge or discharge an electric vehicle according to hourly changing price to sell or buy. The price pattern also varies from day to day. The goal of this work is to schedule the optimal charging/discharging actions so that it can save money.</p>
  
  <p>The state contains past n-step price records, current energy level in battery, hour, etc., like below:</p>
  
  <p><span class="math-container">$$
  s_t = \{ p_{t-5}, p_{t-4}, p_{t-3}, p_{t-2}, p_{t-1}, E_t, t \}
  $$</span></p>
  
  <p>What I'm wondering is whether this is a partial observable situation or not, because the agent can only observe past n-step prices rather than knowing every price at every time step.</p>
  
  <p>Can anyone comment on this issue?</p>
  
  <p>If this is the partial observable situation, is there any simple way to deal with it?</p>
  
Tags: <reinforcement-learning><definitions><dqn>
Title: Is a state that includes only the past n-step price records partially observable?
UserDisplayName: JH Lee
ThreadText: # Is a state that includes only the past n-step price records partially observable?
  
  (Asked by: username_0 on 2019-08-29)
  
  I'm currently working on a project to make an DQN agent that decides whether to charge or discharge an electric vehicle according to hourly changing price to sell or buy. The price pattern also varies from day to day. The goal of this work is to schedule the optimal charging/discharging actions so that it can save money.
  
  The state contains past n-step price records, current energy level in battery, hour, etc., like below:
  
  $$
  s\_t = \{ p\_{t-5}, p\_{t-4}, p\_{t-3}, p\_{t-2}, p\_{t-1}, E\_t, t \}
  $$
  
  What I'm wondering is whether this is a partial observable situation or not, because the agent can only observe past n-step prices rather than knowing every price at every time step.
  
  Can anyone comment on this issue?
  
  If this is the partial observable situation, is there any simple way to deal with it?
  
  **Tags:** reinforcement-learning, definitions, dqn
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-09-01. Score: 1 | Accepted answer
  
  An environment is partially observable if the agent cannot *fully* observe the current state but it only *partially* observes it. More specifically, in *fully* observable MDPs (FOMDPs), the agent knows the current state of the environment, which *can or not* (for example, depending on whether the state is Markov or not) contain all the information required to theoretically take the optimal action. In a *partially* observable MDPs (POMDPs), the state is not available or observable, so the agent might not possess all the required information to theoretically take the optimal action.
  
  For example, in the game of poker, there are hidden and observable cards. In this game, the state of the environment *could* be defined as all the hands of all players, the cards on the table and the next cards that will be drawn from the deck until the end of the round. If a player had access to all this information, then the environment would be fully observable, but, usually, this information is not available, so poker is usually considered a partially observable game.
  
  You *defined* the state as follows
  
  $$s\_t = \{ p\_{t:t-n}, E\_t\}$$
  
  So, if you have access to the past $n$ price records, $p\_{1:n}$, and the energy level, $E\_t$, then your environment is fully observable. However, this does not mean that it is a Markov environment, that is, intuitively, that your current state is sufficient to theoretically take the optimal action.
  
  To conclude, partial observability depends on the definition of state in a specific problem. The concept of Markov property is also related to the concept of partial observability. However, in the case of MDPs, the concept of Markov property has a specific definition: given the current state and an action, the probability of the next state is conditionally independent of all previous states and actions or, more formally
  
  $$
  P(S\_{t+1} \mid S\_t, A\_t) = P(S\_{t+1} \mid S\_t, S\_{t-1}, \dots, S\_1, A\_t, A\_{t-1}, \dots, A\_1)
  $$
  
  where $S\_{t+1}$ is the next state, $S\_t$ the current state, $A\_t$ the current action, $S\_{t-1}, \dots, S\_1$ the previous states and $A\_{t-1}, \dots, A\_1$ the previous actions.
  
  > **Comments:**
  > **username_0**: Thank you for clear explanationI have an additional question. As you said in your answer, to find optimal action it will be needed to give sufficient state information (Markov environment). If that is true, my state representation in the problem explained in my question is not providing sufficient information because it provides 'past n-step price' rather than 'the price on the hour it trades'. I intended to give helpful information to infer the price on the hour because practically the price on that time is not known.  
  > **username_0**: In this case, am I wrongly defining the problem or is it something that reinforcement learning cannot solve?  
  > **username_1**: @JHLee I would ask this as a new question on the website. I would also add details related to your definition of the reward function and possible actions.  
  
  ---
  
  