Answers.Body: <blockquote>
    <p>Does the opponent's turn affect the calculated rewards? </p>
  </blockquote>
  
  <p>Yes, in general it can. Obvious case, in a two player game where the opponent could win or lose on their turn, but has other options.</p>
  
  <blockquote>
    <p>As far as I know, the reward should only be the result of the agent's action right?</p>
  </blockquote>
  
  <p>In a well-defined MDP, the reward should be a stochastic function of the current state and the agent's action. The stochastic part can include any changes due to an opponent player.</p>
  
  <p>If the opponent player is random, or follows a well-defined and fixed policy, then you consider them part of the environment. So this requirement is met technically. The reward does only depend on current state and the agent's action. The actual result may happen on the opponent's turn, but that does not matter. </p>
  
  <p>In a card game where the opponent's cards are hidden and affect their strategy, this may not strictly be the case, because the visible state will not determine the opponent's behaviour. The problem stops being an MDP, and starts being a POMDP. Whether or not that impacts the agent will depend on how much strategy relies on the hidden nature of these cards. In blackjack, there is little impact to not knowing an opponent's cards before they are played out - there is little difference between hidden cards and cards that are randomly determined after the agent plays. So you can get away with pretending it is a normal MDP. In poker, the knowledge of hidden cards is almost everything about the game, so a POMDP or other approach that tracks possible hidden state is required.</p>
  
  <p>Note that learning to defeat a random or expert player is usually not the same as learning to play optimally (unless your expert player is already optimal). For that you may need self-play and an agent which learns both players' policies.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-07-15T16:27:12.017Z
Answers.Text: So if for example the agent's action results in him capturing 0 cards on the table but when the opponent plays, he captures 2. Does this mean the agent failed? Would the reward be -2?
Answers.UserDisplayName: Haytam
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-07-15T16:28:10.120Z
Answers.Text: Another scenario would be the end of the game, where my agent captures 0 cards and the cards left (e.g. 3) on the table go to his opponent (assuming he's the last capturing player), I believe in this case the reward would be -3 to teach the agent that he needs to capture last in the last round?
Answers.UserDisplayName: Haytam
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-07-15T16:32:33.623Z
Answers.Text: @Haytam: If the goal is definitely to capture cards, and not let the opponent do so (i.e. that is all that matters in the game), then you can have rewards that count cards. Otherwise, just +1 for a win, 0 for a draw and -1 for losing at the *end of the game* should be fine, and is a standard approach for RL playing classic 2-player games.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-07-15T16:34:30.427Z
Answers.Text: The goal is to capture cards, but efficiently. If the agent has the choice to capture 1 card or 3 sequential cards, I want him to choose the 3 sequential cards, that's why I want my reward to be the number of cards eaten, but I want to be sure about the opponent's turn, because I don't think the opponent's turn has anything to do with my agent's action right?
Answers.UserDisplayName: Haytam
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-07-15T19:04:22.673Z
Answers.Text: @Haytam: The opponent's turn is definitely important to the agent, and you should find a way to include its effects. I don't know the card game you are modelling, so it *might* be reasonable to ignore how many cards the opponent captures. However, my gut instinct says that is not the case. You mention "sequential cards" as a desirable outcome - this implies there is more to this game than just total number of captures (that's how I understand the term "sequential", unless you just mean "total"). So using captures as a reward signal would be inaccurate. Winning at the end always matters though.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-07-15T20:28:59.047Z
Answers.Text: If you're interesting about the game: https://www.pagat.com/fishing/ronda.html
  What I mean by sequential is that if on the table there are cards 1, 2, 3, 4 and the agent has the cards 1 and 3, I want him to capture with the card 1 so that he gets 1>2>3>4 rather than capturing with 3 which will only get him 3>4
Answers.UserDisplayName: Haytam
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-07-15T20:30:58.837Z
Answers.Text: I apologize for the long texts, is there any way I can contact you outside of here? If you have the time of course.
Answers.UserDisplayName: Haytam
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-07-15T20:45:56.653Z
Answers.Text: @Haytam: If you have further questions, I would prefer you ask new ones on the site. However, just from your brief description, I am even more certain that you want end-result win/draw/lose for reward. Any rewards that count captures seem to me to be more like heuristics, and not really suitable for a reward signal.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-07-15T16:05:53.023Z
Answers.LastActivityDate: 2018-07-15T16:30:40.507Z
Answers.LastEditDate: 2018-07-15T16:30:40.507Z
Answers.UserDisplayName: Neil Slater
Body: <p>I made an engine for a 2 players card game and now I am trying to make an environment similar to OpenAI Gym envs, to ease out the training.</p>
  
  <p>I fail to understand this thing however:  </p>
  
  <ol>
  <li>If I use <code>step(agentAction)</code>, I play the agent's turn in the game,
  calculate the reward.</li>
  <li>Play the opponent's turn <em>(which will be either a random AI or a rule-based one)</em>.</li>
  </ol>
  
  <p><strong>Question:</strong><br>
  Does the opponent's turn affect the calculated rewards? As far as I know, the reward should only be the result of the agent's action right?</p>
  
  <p>Thank you.</p>
  
Tags: <reinforcement-learning><dqn><open-ai>
Title: Can the opponent's turn affect the reward for a DQN agent action?
UserDisplayName: Haytam
ThreadText: # Can the opponent's turn affect the reward for a DQN agent action?
  
  (Asked by: username_0 on 2018-07-15)
  
  I made an engine for a 2 players card game and now I am trying to make an environment similar to OpenAI Gym envs, to ease out the training.
  
  I fail to understand this thing however:
  
  1. If I use `step(agentAction)`, I play the agent's turn in the game,
     calculate the reward.
  2. Play the opponent's turn *(which will be either a random AI or a rule-based one)*.
  
  **Question:**  
  Does the opponent's turn affect the calculated rewards? As far as I know, the reward should only be the result of the agent's action right?
  
  Thank you.
  
  **Tags:** reinforcement-learning, dqn, open-ai
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-07-15. Score: 3
  
  > Does the opponent's turn affect the calculated rewards?
  
  Yes, in general it can. Obvious case, in a two player game where the opponent could win or lose on their turn, but has other options.
  
  > As far as I know, the reward should only be the result of the agent's action right?
  
  In a well-defined MDP, the reward should be a stochastic function of the current state and the agent's action. The stochastic part can include any changes due to an opponent player.
  
  If the opponent player is random, or follows a well-defined and fixed policy, then you consider them part of the environment. So this requirement is met technically. The reward does only depend on current state and the agent's action. The actual result may happen on the opponent's turn, but that does not matter.
  
  In a card game where the opponent's cards are hidden and affect their strategy, this may not strictly be the case, because the visible state will not determine the opponent's behaviour. The problem stops being an MDP, and starts being a POMDP. Whether or not that impacts the agent will depend on how much strategy relies on the hidden nature of these cards. In blackjack, there is little impact to not knowing an opponent's cards before they are played out - there is little difference between hidden cards and cards that are randomly determined after the agent plays. So you can get away with pretending it is a normal MDP. In poker, the knowledge of hidden cards is almost everything about the game, so a POMDP or other approach that tracks possible hidden state is required.
  
  Note that learning to defeat a random or expert player is usually not the same as learning to play optimally (unless your expert player is already optimal). For that you may need self-play and an agent which learns both players' policies.
  
  > **Comments:**
  > **username_0**: So if for example the agent's action results in him capturing 0 cards on the table but when the opponent plays, he captures 2. Does this mean the agent failed? Would the reward be -2?  
  > **username_0**: Another scenario would be the end of the game, where my agent captures 0 cards and the cards left (e.g. 3) on the table go to his opponent (assuming he's the last capturing player), I believe in this case the reward would be -3 to teach the agent that he needs to capture last in the last round?  
  > **username_1**: @Haytam: If the goal is definitely to capture cards, and not let the opponent do so (i.e. that is all that matters in the game), then you can have rewards that count cards. Otherwise, just +1 for a win, 0 for a draw and -1 for losing at the *end of the game* should be fine, and is a standard approach for RL playing classic 2-player games.  
  > **username_0**: The goal is to capture cards, but efficiently. If the agent has the choice to capture 1 card or 3 sequential cards, I want him to choose the 3 sequential cards, that's why I want my reward to be the number of cards eaten, but I want to be sure about the opponent's turn, because I don't think the opponent's turn has anything to do with my agent's action right?  
  > **username_1**: @Haytam: The opponent's turn is definitely important to the agent, and you should find a way to include its effects. I don't know the card game you are modelling, so it *might* be reasonable to ignore how many cards the opponent captures. However, my gut instinct says that is not the case. You mention "sequential cards" as a desirable outcome - this implies there is more to this game than just total number of captures (that's how I understand the term "sequential", unless you just mean "total"). So using captures as a reward signal would be inaccurate. Winning at the end always matters though.  
  > **username_0**: If you're interesting about the game: https://www.pagat.com/fishing/ronda.html What I mean by sequential is that if on the table there are cards 1, 2, 3, 4 and the agent has the cards 1 and 3, I want him to capture with the card 1 so that he gets 1>2>3>4 rather than capturing with 3 which will only get him 3>4  
  > **username_0**: I apologize for the long texts, is there any way I can contact you outside of here? If you have the time of course.  
  > **username_1**: @Haytam: If you have further questions, I would prefer you ask new ones on the site. However, just from your brief description, I am even more certain that you want end-result win/draw/lose for reward. Any rewards that count captures seem to me to be more like heuristics, and not really suitable for a reward signal.  
  
  ---
  
  