Answers.Body: <p>The code you reference is not part of the learning agent. It is part of:</p>
  
  <pre><code>class BlackjackEnv(gym.Env):
  </code></pre>
  
  <p>If, as in this case, the environment is provided entirely by software simulation, it is absolutely necessary for it to include a full working model of all state transitions and rewards. That is independent of whether any hidden state makes the problem harder. </p>
  
  <blockquote>
    <p>Why is it ok to calculate the reward based on a hidden state?</p>
  </blockquote>
  
  <p>In the case of Blackjack, this can be treated not as a hidden state that would affect the outcome if only known, but as randomness in the environment over which the agent has no control. Critically, the dealer has no options to behave differently depending on the unknown card, and the dealer's eventual score is entirely unaffected by the player's earlier choices.</p>
  
  <p>It is a subtle difference. If you applied the same environment rules to Poker, where an opponent could behave differently depending on this hidden knowledge, then a simple MDP model is not enough theory to result in an optimal solution. In that case, you would need to look into <a href="https://en.wikipedia.org/wiki/Partially_observable_Markov_decision_process" rel="nofollow noreferrer">Partially Observable MDPs (POMDPs)</a>. Note this would not affect reward calculation in the environment, just the choices of which agent type to use. If you are just learning RL, you probably don't know of any algorithms that could solve this yet.</p>
  
  <p>In practice, a lot of problems are somewhere between a classic MDP and a POMDP - they contain elements which, if the agent could know them, may allow it to achieve a higher expected reward. In many cases though, these elements can either be treated as random (as here in Blackjack) and thus the system is still theoretically an MDP, or they have a very small effect on the optimal policy, so can be ignored for practical purposes (e.g. think of all the physical details in a real cart pole balancing system - friction, temperature, flexing motions, etc).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-11-22T09:49:58.210Z
Answers.LastActivityDate: 2020-04-03T21:03:04.793Z
Answers.LastEditDate: 2020-04-03T21:03:04.793Z
Answers.UserDisplayName: Neil Slater
Body: <p>I'm looking at <a href="https://github.com/openai/gym/blob/master/gym/envs/toy_text/blackjack.py" rel="nofollow noreferrer">this source code</a>, where the reward is calculated with</p>
  
  <pre><code>reward = cmp(score(self.player), score(self.dealer))
  </code></pre>
  
  <p>Why is it ok to calculate the reward based on a hidden state?</p>
  
  <p>A player only sees the dealer's first card.</p>
  
  <pre><code>self.dealer[0]
  </code></pre>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-11-22T09:29:06.050Z
Comments.Text: The reward is just a number. It tells nothing more than that. You, as a teacher or programmer, can model the reward function as you well think.
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><rewards>
Title: Why is it ok to calculate the reward based on a hidden state?
UserDisplayName: Alex Mistro
ThreadText: # Why is it ok to calculate the reward based on a hidden state?
  
  (Asked by: username_0 on 2018-11-22)
  
  I'm looking at [this source code](https://github.com/openai/gym/blob/master/gym/envs/toy_text/blackjack.py), where the reward is calculated with
  
  ```
  reward = cmp(score(self.player), score(self.dealer))
  
  ```
  
  Why is it ok to calculate the reward based on a hidden state?
  
  A player only sees the dealer's first card.
  
  ```
  self.dealer[0]
  
  ```
  
  **Tags:** reinforcement-learning, rewards
  
  > **Comments:**
  > **username_1**: The reward is just a number. It tells nothing more than that. You, as a teacher or programmer, can model the reward function as you well think.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2018-11-22. Score: 1 | Accepted answer
  
  The code you reference is not part of the learning agent. It is part of:
  
  ```
  class BlackjackEnv(gym.Env):
  
  ```
  
  If, as in this case, the environment is provided entirely by software simulation, it is absolutely necessary for it to include a full working model of all state transitions and rewards. That is independent of whether any hidden state makes the problem harder.
  
  > Why is it ok to calculate the reward based on a hidden state?
  
  In the case of Blackjack, this can be treated not as a hidden state that would affect the outcome if only known, but as randomness in the environment over which the agent has no control. Critically, the dealer has no options to behave differently depending on the unknown card, and the dealer's eventual score is entirely unaffected by the player's earlier choices.
  
  It is a subtle difference. If you applied the same environment rules to Poker, where an opponent could behave differently depending on this hidden knowledge, then a simple MDP model is not enough theory to result in an optimal solution. In that case, you would need to look into [Partially Observable MDPs (POMDPs)](https://en.wikipedia.org/wiki/Partially_observable_Markov_decision_process). Note this would not affect reward calculation in the environment, just the choices of which agent type to use. If you are just learning RL, you probably don't know of any algorithms that could solve this yet.
  
  In practice, a lot of problems are somewhere between a classic MDP and a POMDP - they contain elements which, if the agent could know them, may allow it to achieve a higher expected reward. In many cases though, these elements can either be treated as random (as here in Blackjack) and thus the system is still theoretically an MDP, or they have a very small effect on the optimal policy, so can be ignored for practical purposes (e.g. think of all the physical details in a real cart pole balancing system - friction, temperature, flexing motions, etc).
  
  
  ---
  
  