Answers.Body: <p>It is hard to say for certain without knowing full details and results of experiments.</p>
  <p>However, if the game allows for splitting decisions up, it will likely be better for the agent to take advantage of extra knowledge of the value of any previously hidden card just taken from the draw pile.</p>
  <p>In general, if each player decision is taken sequentially, resulting in changes to state, then it is a separate action on a separate time step according to the MDP theoretical model used in reinforcement learning (RL). You might want to describe/notate the time steps differently so that they match how the game play proceeds. However, for the purposes of RL, each decision point should be on a new time step, and should result in a new state, new value estimates etc.</p>
  <p>Similarly, whether or not the current choice is the player's first card or second card to be drawn needs to be part of the state. This detail of the state <em>might</em> already be covered by the number of cards in the player's hand, if logically the number of cards is always the same at each stage. However, if hand size can vary for other reasons, it is worth adding an explicit flag for &quot;first draw choice&quot; or similar so that the agent can use the information.</p>
  <p>You have some freedom for encoding the action space. If drawing cards is the only possible action in this game at all stages, then a binary output vector of 1 + (number_of_players - 1) dimensions would be suitable. Other encodings may work well too, it depends if there is any logical structure to the choices or some derived data that encodes useful game information.</p>
  <p>It may be useful to arrange the action choices so that the index for drawing from each player's discard pile is considered <em>relatively</em> to the current player's turn. That is, instead of actions being arranged <span class="math-container">$[draw, discard P1, discard P3, discard P4, discard P5]$</span> for P2, they would be arranged <span class="math-container">$[draw, discard P3, discard P4, discard P5, discard P1]$</span> and for P3 would be different: <span class="math-container">$[draw, discard P4, discard P5, discard P1, discard P2]$</span> . . . that would inherently allow for the cyclical nature of turns. State representation would need to similarly rotate knowledge about each player to match this. You might not need to do this, but I would recommend it for games where there is a lot of common logic regarding action choices relative to turn position that you could take advantage of. The opposite would apply (and you would use absolute player positions) if there were important differences throughout the game between being P1, P2, P3 etc.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-28T05:38:55.880Z
Answers.Text: Thanks! There are other actions after cards are drawn, namely selection and placement of a card (the encoding for which seems like a separate problem as the cards are places in a growing tiled area, similar to dominos). The relative ordering also makes sense.
Answers.UserDisplayName: thinkski
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-27T15:31:05.373Z
Answers.LastActivityDate: 2020-12-27T17:11:53.560Z
Answers.LastEditDate: 2020-12-27T17:11:53.560Z
Answers.UserDisplayName: Neil Slater
Body: <p>I have a card game where on a player's turn, the player sequentially draws two cards. Each card may be drawn from another player's discard stack (face up), or from the deck (face down).</p>
  <p>Thinking how to encode this into an action space, I could naively assume the two draws are independent. The action space would simply be a binary vector of 2 * (1 + (number_of_players - 1)), which I could post-filter to limit for empty draw piles (and can't draw from own pile).</p>
  <p>However, when playing the game myself, I noticed that it's sometimes advantageous to draw the initial card from the deck, then select the draw pile for the second card based on the value of the first one drawn. But how would this be encoded into an action space? Would it be better to think of these are two separate actions, even thought they are part of the same &quot;turn&quot;?</p>
  
Tags: <reinforcement-learning>
Title: RL: Encoding action conditioned on previous action
UserDisplayName: thinkski
ThreadText: # RL: Encoding action conditioned on previous action
  
  (Asked by: username_0 on 2020-12-27)
  
  I have a card game where on a player's turn, the player sequentially draws two cards. Each card may be drawn from another player's discard stack (face up), or from the deck (face down).
  
  Thinking how to encode this into an action space, I could naively assume the two draws are independent. The action space would simply be a binary vector of 2 \* (1 + (number\_of\_players - 1)), which I could post-filter to limit for empty draw piles (and can't draw from own pile).
  
  However, when playing the game myself, I noticed that it's sometimes advantageous to draw the initial card from the deck, then select the draw pile for the second card based on the value of the first one drawn. But how would this be encoded into an action space? Would it be better to think of these are two separate actions, even thought they are part of the same "turn"?
  
  **Tags:** reinforcement-learning
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-12-27. Score: 2 | Accepted answer
  
  It is hard to say for certain without knowing full details and results of experiments.
  
  However, if the game allows for splitting decisions up, it will likely be better for the agent to take advantage of extra knowledge of the value of any previously hidden card just taken from the draw pile.
  
  In general, if each player decision is taken sequentially, resulting in changes to state, then it is a separate action on a separate time step according to the MDP theoretical model used in reinforcement learning (RL). You might want to describe/notate the time steps differently so that they match how the game play proceeds. However, for the purposes of RL, each decision point should be on a new time step, and should result in a new state, new value estimates etc.
  
  Similarly, whether or not the current choice is the player's first card or second card to be drawn needs to be part of the state. This detail of the state *might* already be covered by the number of cards in the player's hand, if logically the number of cards is always the same at each stage. However, if hand size can vary for other reasons, it is worth adding an explicit flag for "first draw choice" or similar so that the agent can use the information.
  
  You have some freedom for encoding the action space. If drawing cards is the only possible action in this game at all stages, then a binary output vector of 1 + (number\_of\_players - 1) dimensions would be suitable. Other encodings may work well too, it depends if there is any logical structure to the choices or some derived data that encodes useful game information.
  
  It may be useful to arrange the action choices so that the index for drawing from each player's discard pile is considered *relatively* to the current player's turn. That is, instead of actions being arranged $[draw, discard P1, discard P3, discard P4, discard P5]$ for P2, they would be arranged $[draw, discard P3, discard P4, discard P5, discard P1]$ and for P3 would be different: $[draw, discard P4, discard P5, discard P1, discard P2]$ . . . that would inherently allow for the cyclical nature of turns. State representation would need to similarly rotate knowledge about each player to match this. You might not need to do this, but I would recommend it for games where there is a lot of common logic regarding action choices relative to turn position that you could take advantage of. The opposite would apply (and you would use absolute player positions) if there were important differences throughout the game between being P1, P2, P3 etc.
  
  > **Comments:**
  > **username_0**: Thanks! There are other actions after cards are drawn, namely selection and placement of a card (the encoding for which seems like a separate problem as the cards are places in a growing tiled area, similar to dominos). The relative ordering also makes sense.  
  
  ---
  
  