Answers.Body: <p>If your opponent has fixed knowledge (it doesn't learn), then the next state after your agent did an action is the state when your turn is back. So the actions of other players are considered as an environment reaction to your actions.</p>
  
  <p>But if your opponent can learn, you may create a <a href="https://arxiv.org/pdf/1711.00832.pdf" rel="nofollow noreferrer">Multi-agent Reinforcement Learning</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-10T09:26:25.487Z
Answers.Text: Thank you. But can I ask what difference it makes if the opponent can or cannot learn? I'm not sure why whether they learn or not makes any difference to *this* agent's perspective? The reason I ask is for the Q-learning update rule where you use $maxaQ(S′,a)$I just had to be sure $S'$ is the State the agent encounters when it's their next turn, not immediately after performing its action.  Thank you for the link to the paper, I will read with interest.
Answers.UserDisplayName: BigBadMe
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-10T09:54:52.010Z
Answers.Text: In multi-agent RL, your opponent is not like opponent in the real game, your agent can learn from your opponent's experience. So in the match, your agent is trained on both side (it's used to improve the training process). But in normal RL games, $S'$ is the states that your agent received (on its turn).
Answers.UserDisplayName: malioboro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-10T10:21:11.367Z
Answers.Text: @BigBadMe I would not say that the distinguishing factor is whether or not the opponent can learn. I'd say the distinguishing factor is whether or not **your learning algorithm is aware of the opponent's ability to learn**. If you're trying to apply a single-agent learning algorithm like vanilla $Q$-learning, that algorithm is not even aware of the opponent's existence, so you have to treat the opponent's actions as a part of the environment. Some dedicated multi-agent RL algorithms may be "opponent aware".
Answers.UserDisplayName: Dennis Soemers
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-10T02:06:54.060Z
Answers.LastActivityDate: 2019-02-10T09:56:31.720Z
Answers.LastEditDate: 2019-02-10T09:56:31.720Z
Answers.UserDisplayName: malioboro
Body: <p>I'm using Q-learning to train an agent to play a board game (e.g. chess, draughts or go). </p>
  
  <p>The agent takes an action while in state <span class="math-container">$S$</span>, but then what is the next state (that is,  <span class="math-container">$S'$</span>)? Is <span class="math-container">$S'$</span> now the board with the piece moved as a result of taking the action, or is <span class="math-container">$S'$</span> the state the agent encounters after the other player has performed his action (i.e. it's this agent's turn again)?</p>
  
Tags: <reinforcement-learning><q-learning>
Title: What is the next state for a two-player board game?
UserDisplayName: BigBadMe
ThreadText: # What is the next state for a two-player board game?
  
  (Asked by: username_0 on 2019-02-09)
  
  I'm using Q-learning to train an agent to play a board game (e.g. chess, draughts or go).
  
  The agent takes an action while in state $S$, but then what is the next state (that is, $S'$)? Is $S'$ now the board with the piece moved as a result of taking the action, or is $S'$ the state the agent encounters after the other player has performed his action (i.e. it's this agent's turn again)?
  
  **Tags:** reinforcement-learning, q-learning
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-02-10. Score: 2 | Accepted answer
  
  If your opponent has fixed knowledge (it doesn't learn), then the next state after your agent did an action is the state when your turn is back. So the actions of other players are considered as an environment reaction to your actions.
  
  But if your opponent can learn, you may create a [Multi-agent Reinforcement Learning](https://arxiv.org/pdf/1711.00832.pdf)
  
  > **Comments:**
  > **username_0**: Thank you. But can I ask what difference it makes if the opponent can or cannot learn? I'm not sure why whether they learn or not makes any difference to *this* agent's perspective? The reason I ask is for the Q-learning update rule where you use $maxaQ(S′,a)$I just had to be sure $S'$ is the State the agent encounters when it's their next turn, not immediately after performing its action.  Thank you for the link to the paper, I will read with interest.  
  > **username_1**: In multi-agent RL, your opponent is not like opponent in the real game, your agent can learn from your opponent's experience. So in the match, your agent is trained on both side (it's used to improve the training process). But in normal RL games, $S'$ is the states that your agent received (on its turn).  
  > **username_2**: @username_0 I would not say that the distinguishing factor is whether or not the opponent can learn. I'd say the distinguishing factor is whether or not **your learning algorithm is aware of the opponent's ability to learn**. If you're trying to apply a single-agent learning algorithm like vanilla $Q$-learning, that algorithm is not even aware of the opponent's existence, so you have to treat the opponent's actions as a part of the environment. Some dedicated multi-agent RL algorithms may be "opponent aware".  
  
  ---
  
  