Body: <p>Suppose there are two players in my zero-sum game and they play in a row like chess. And I want to learn the policy function using the REINFORCE algorithm.</p>
  <p>I have doubts about passing reward values in the episode trajectory. If there is a single agent, then it is straightforward to pass the reward values based on the action performed by the agent in a particular state. But, in the case of multiple players, action by any player <span class="math-container">$k$</span> may need the updation of reward for any other player <span class="math-container">$\ell$</span>. In such cases, I have doubts about passing rewards for player <span class="math-container">$\ell$</span>.</p>
  <p>Suppose, in the state <span class="math-container">$s_0$</span>, player 1 did an action <span class="math-container">$a_0$</span> and causes the reward of <span class="math-container">$r_1$</span> to player 1 and -1 to player 2 and leads to state <span class="math-container">$s_1$</span>. Then while it's turn, player 2 performs some action <span class="math-container">$a_1$</span> on state <span class="math-container">$s_1$</span> which caused a reward of <span class="math-container">$r_2$</span> to it. Then which trajectory should I need to pass among the following?</p>
  <ol>
  <li><span class="math-container">$s_0a_0r_1s_1a_1r_2 \cdots$</span></li>
  <li><span class="math-container">$s_0a_0r_1s_1a_1(r_2-1) \cdots$</span></li>
  </ol>
  <p>I am guessing the second one. But my doubt is that the reward <span class="math-container">$r_2$</span> is due to the action of player 2 and <span class="math-container">$r_2-1$</span> cannot be viewed as an immediate reward since <span class="math-container">$-1$</span> is due to the action of player 1 in the previous time step. How to pass the reward values in this case?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-10T17:24:41.203Z
Comments.Text: Let me try to understand if I understood correctly what you're trying to do. You're trying to apply REINFORCE to a two-player game. Both players don't take an action at each step, but you make one player take an action at step $t$ and the next player at step $t+1$, and so on. From your attempt to define the reward $r_2 - 1$, it seems that you're also trying to penalise the second player based on the past actions of player 1. But why don't you also penalise player $1$? Are you trying to find a policy only for player 1? It's not clear what kind of policy you're trying to find here...
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-10T17:27:32.963Z
Comments.Text: ... because it seems that you're taking trajectories that take into account the moves of both players in order to update the policy, but then you only seem to penalise one player. I am not sure if this approach makes sense. So, maybe you should try to clarify what your policy is supposed to do.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-10T22:35:28.447Z
Comments.Text: **But why don't you also penalise player 1?** It is just an example. Player 1 may also get penalized in later steps due to the action of player 2. **Are you trying to find a policy only for player 1?**. No, I am not trying to find the policy for a single player. I want to update the same NN for all players. @nbro
Comments.UserDisplayName: hanugm
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-10T22:45:25.400Z
Comments.Text: Important: Is the game a zero-sum game? Chess is, for example, it has simple "I win, you lose" combination. If it is a zero-sum game, then any collection of "points" along the way is secondary. You can treat gaing game assets (such as cash in monopoly) as a reward, but it may make more sense to treat it as state information, so you can use the zero-sum end result to drive the rewards, if all that matters is who wins at the end.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-10T22:52:00.570Z
Comments.Text: @NeilSlater Do you mean I should allot/fill reward values in the episode trajectory after the completion of the chess game based on the winner/loser?
Comments.UserDisplayName: hanugm
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-10T22:53:48.683Z
Comments.Text: That's not what I said, I was asking a for clarification about your scenario. However, that may be the best answer if you really do have a zero-sum game, and don't need to do any reward shaping.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-10T23:00:17.880Z
Comments.Text: Yeah, the game is a zero-sum game. I updated it now. I don't know prior that `zero-sum` property affects the way of providing rewards in the episode trajectory @NeilSlater. If possible, please provide or tell me how to fill rewards after the completion of the game in the trajectory. Should I feel 1 for all actions by the winner and -1 for all actions by the loser in the trajectory?
Comments.UserDisplayName: hanugm
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-11T07:58:03.493Z
Comments.Text: @hanugm: zero-sum implies $r_{a,t} = -r_{b.t}$, which most importantly *allows* you to have a single model for both players (one of your stated goals). Technically you could combine player policy models for non-zero-sum games into a multi-headed network - there are a few different ways to model game playing. I am not sure of the consequences for training a single model using REINFORCE this way.
Comments.UserDisplayName: Neil Slater
Tags: <reinforcement-learning><deep-rl><rewards><reinforce>
Title: How to pass the rewards in zero-sum multiplayer context when using REINFORCE?
UserDisplayName: hanugm
ThreadText: # How to pass the rewards in zero-sum multiplayer context when using REINFORCE?
  
  (Asked by: username_0 on 2022-01-10)
  
  Suppose there are two players in my zero-sum game and they play in a row like chess. And I want to learn the policy function using the REINFORCE algorithm.
  
  I have doubts about passing reward values in the episode trajectory. If there is a single agent, then it is straightforward to pass the reward values based on the action performed by the agent in a particular state. But, in the case of multiple players, action by any player $k$ may need the updation of reward for any other player $\ell$. In such cases, I have doubts about passing rewards for player $\ell$.
  
  Suppose, in the state $s\_0$, player 1 did an action $a\_0$ and causes the reward of $r\_1$ to player 1 and -1 to player 2 and leads to state $s\_1$. Then while it's turn, player 2 performs some action $a\_1$ on state $s\_1$ which caused a reward of $r\_2$ to it. Then which trajectory should I need to pass among the following?
  
  1. $s\_0a\_0r\_1s\_1a\_1r\_2 \cdots$
  2. $s\_0a\_0r\_1s\_1a\_1(r\_2-1) \cdots$
  
  I am guessing the second one. But my doubt is that the reward $r\_2$ is due to the action of player 2 and $r\_2-1$ cannot be viewed as an immediate reward since $-1$ is due to the action of player 1 in the previous time step. How to pass the reward values in this case?
  
  **Tags:** reinforcement-learning, deep-rl, rewards, reinforce
  
  > **Comments:**
  > **username_1**: Let me try to understand if I understood correctly what you're trying to do. You're trying to apply REINFORCE to a two-player game. Both players don't take an action at each step, but you make one player take an action at step $t$ and the next player at step $t+1$, and so on. From your attempt to define the reward $r_2 - 1$, it seems that you're also trying to penalise the second player based on the past actions of player 1. But why don't you also penalise player $1$? Are you trying to find a policy only for player 1? It's not clear what kind of policy you're trying to find here...  
  > **username_1**: ... because it seems that you're taking trajectories that take into account the moves of both players in order to update the policy, but then you only seem to penalise one player. I am not sure if this approach makes sense. So, maybe you should try to clarify what your policy is supposed to do.  
  > **username_0**: **But why don't you also penalise player 1?** It is just an example. Player 1 may also get penalized in later steps due to the action of player 2. **Are you trying to find a policy only for player 1?**. No, I am not trying to find the policy for a single player. I want to update the same NN for all players. @username_1  
  > **username_2**: Important: Is the game a zero-sum game? Chess is, for example, it has simple "I win, you lose" combination. If it is a zero-sum game, then any collection of "points" along the way is secondary. You can treat gaing game assets (such as cash in monopoly) as a reward, but it may make more sense to treat it as state information, so you can use the zero-sum end result to drive the rewards, if all that matters is who wins at the end.  
  > **username_0**: @username_2 Do you mean I should allot/fill reward values in the episode trajectory after the completion of the chess game based on the winner/loser?  
  > **username_2**: That's not what I said, I was asking a for clarification about your scenario. However, that may be the best answer if you really do have a zero-sum game, and don't need to do any reward shaping.  
  > **username_0**: Yeah, the game is a zero-sum game. I updated it now. I don't know prior that `zero-sum` property affects the way of providing rewards in the episode trajectory @username_2. If possible, please provide or tell me how to fill rewards after the completion of the game in the trajectory. Should I feel 1 for all actions by the winner and -1 for all actions by the loser in the trajectory?  
  > **username_2**: @username_0: zero-sum implies $r_{a,t} = -r_{b.t}$, which most importantly *allows* you to have a single model for both players (one of your stated goals). Technically you could combine player policy models for non-zero-sum games into a multi-headed network - there are a few different ways to model game playing. I am not sure of the consequences for training a single model using REINFORCE this way.  
  
  ---
  
  ## 0 answers
  
  