Answers.Body: <p>I would recommend having the reward as the increase to score caused by adding the card in the chosen location. You <em>could</em> optionally include calculations based on partial rows for pairs, three-of-a-kind etc. I do not think you should grant a different final reward for the end score, and definitely not some large difference, as the gradient for a large difference may swamp the networks (unless you also drop the learning rate, but then the smaller early values will not get learned from).</p>
  <p>I do not think that your focus should be on clever manipulation of reward signal. It is likely that looking at other things related to this problem will give you better impact.</p>
  <p>First you should look at your state representation. If you are feeding the <code>int8</code> values directly to a neural network, then this is very unlikely to work. Instead, you should use a short vector for each card you want to represent - of either 6 or 15 elements. You should be one-hot-encoding the colour for each position. You may find one-hot-encoding the value also helps, although that is trickier - if you don't one-hot encode, then you should scale the value e.g. <span class="math-container">$(v - 5)/3$</span> to keep it in range for best neural network learning.</p>
  <p>The card in hand, about to be placed, should be part of the state. So far, before any feature engineering, that would give you a vector of at least 26 * 6 = 156 dimensions, but you could take that further.</p>
  <p>If possible, use <a href="https://stats.stackexchange.com/questions/411932/reinforcement-learning-afterstate-and-afterstate-value-functions">afterstates</a> instead of enumerated actions. That may not be possible if you are using a library that expects enumerated actions. You probably won't find a version of Actor-Critic with afterstates because the Actor would need to express a probability distribution over the afterstates, which is awkward to do. You may be able to set the Actor to use enumerated actions and the Critic to score advantage based on afterstates though.</p>
  <p>To help the agent, do some feature engineering to add summaries of each possible line to the state (so a summary for each of the 12 lines). One important feature will be whether that line is complete or not - if it is complete, no more reward is possible from that line (perhaps a scaled &quot;how many spaces left&quot; counter would work here). Other features that could be useful are a &quot;bag of colours&quot; and &quot;bag of values&quot; i.e. one-hot-encoded list of current variation in that row - such a summary makes it easier for the NN to tell whether an x-of-a-kind or flush or straight are possible.</p>
  <p>In theory you could also pre-calculate the immediate reward from placing the card in each position, making that part of the state (another 25 features). That is so the agent is not trying to predict this value - it is hard for the NN, but easy for the game engine and not &quot;cheating&quot; at the game by e.g. peeking at the next card. That should allow the NN to use more of its capacity for planning ahead and statistically gambling on what might be available in future, which is perhaps the more interesting aspect of this game.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-04-26T20:41:44.547Z
Answers.LastActivityDate: 2022-04-26T20:50:23.840Z
Answers.LastEditDate: 2022-04-26T20:50:23.840Z
Answers.UserDisplayName: Neil Slater
Body: <p>I'm quite new to reinforcement learning. I've been training the model for the following problem but the mean reward is stuck.</p>
  <ul>
  <li>In a 5 by 5 board, each position can contain a card with a color (0-4) and a value (0-9).</li>
  <li>Some initial cards, all distinct from each other, may be on the board.</li>
  <li>In each round, a card not on the board is chosen with uniform distribution.</li>
  <li>The player has to place this card in an empty position before the next card is chosen.</li>
  <li>When all positions are filled, the game ends and the final score is the sum of all points of each row, each column, and each of the two diagonals, where the points are calculated as follows (order does not matter, e.g. AAABB=ABABA, 12345=23415):</li>
  </ul>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th style="text-align: center;">Combination</th>
  <th style="text-align: center;">Points</th>
  <th style="text-align: center;">Colors</th>
  <th style="text-align: center;">Values</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td style="text-align: center;">Straight flush</td>
  <td style="text-align: center;">30</td>
  <td style="text-align: center;">All identical</td>
  <td style="text-align: center;">All consecutive</td>
  </tr>
  <tr>
  <td style="text-align: center;">Straight</td>
  <td style="text-align: center;">15</td>
  <td style="text-align: center;">Not all identical</td>
  <td style="text-align: center;">All consecutive</td>
  </tr>
  <tr>
  <td style="text-align: center;">Flush</td>
  <td style="text-align: center;">14</td>
  <td style="text-align: center;">All identical</td>
  <td style="text-align: center;">Not all consecutive</td>
  </tr>
  <tr>
  <td style="text-align: center;">Five of a kind</td>
  <td style="text-align: center;">28</td>
  <td style="text-align: center;"></td>
  <td style="text-align: center;">AAAAA</td>
  </tr>
  <tr>
  <td style="text-align: center;">Four of a kind</td>
  <td style="text-align: center;">16</td>
  <td style="text-align: center;"></td>
  <td style="text-align: center;">AAAAB</td>
  </tr>
  <tr>
  <td style="text-align: center;">Full house</td>
  <td style="text-align: center;">10</td>
  <td style="text-align: center;"></td>
  <td style="text-align: center;">AAABB</td>
  </tr>
  <tr>
  <td style="text-align: center;">Three of a kind</td>
  <td style="text-align: center;">6</td>
  <td style="text-align: center;"></td>
  <td style="text-align: center;">AAABC</td>
  </tr>
  <tr>
  <td style="text-align: center;">Two pairs</td>
  <td style="text-align: center;">3</td>
  <td style="text-align: center;"></td>
  <td style="text-align: center;">AABBC</td>
  </tr>
  <tr>
  <td style="text-align: center;">One pair</td>
  <td style="text-align: center;">1</td>
  <td style="text-align: center;"></td>
  <td style="text-align: center;">AABCD</td>
  </tr>
  <tr>
  <td style="text-align: center;">Nothing</td>
  <td style="text-align: center;">0</td>
  <td style="text-align: center;"></td>
  <td style="text-align: center;"></td>
  </tr>
  </tbody>
  </table>
  </div>
  <p>The goal is of course to get a score as high as possible.</p>
  <p>Question:
  <strong>How should I write the reward function to teach the agent the rules?</strong></p>
  <p>I used invalid action masking, so no invalid actions can be taken. However, it's still hard for high-score combinations to randomly show up. I tried setting reward to be the points possibly added due to the placed card (like if the placed card can form a pair with another one in the same row, add some points based on how much point a pair is rewarded in the final score), and reward 100*(final score) at the end, hoping the agent to learn that the final score is the most important, but the mean reward was not improving even after 2 million timesteps.</p>
  <p>I use <a href="https://sb3-contrib.readthedocs.io/en/master/modules/ppo_mask.html" rel="nofollow noreferrer">MaskablePPO</a> with MlpPolicy (alias of MaskableActorCritic) from sb3-contrib. I represent each card as an int8. The upper 4 bits representing the color and the lower 4 bits represent the value.</p>
  <p>The observation space I use consists of the card to be placed, the cards on the board, and all cards not on the board. I'm wondering if I should include cards not present in the observation space. I tried to exclude it but it does not change the results much.</p>
  <p>Apart from direct answers, any pointers/reference would be much appreciated.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-04-26T07:25:24.113Z
Comments.Text: Is it a game that you can links the rules for? Your description is fine as-is, but the scoring rules may allow someone to try an experiment. One important detail: Is it possible to calculate a score for the incomplete board using the standard rules? As you have a large state space, and a policy network, I am assuming you are using a neural network. It may be worth explaining what you have tried there, and how you convert state to the representation used by the network. For instance, putting the raw 0-9 values in grid positions to feed into a CNN may not work well.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-04-26T17:00:30.457Z
Comments.Text: @NeilSlater I updated some info. I am not using a CNN tho, because stable-baselines 3 only allows CNN to work with images. I'm using an actor critic policy.
Comments.UserDisplayName: durianice
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-04-26T17:43:19.983Z
Comments.Text: Thanks for the updates. Are the scores assigned according to strict sequence match ie. a straight has to have values in order in the horizontal (left to right) or vertical (top to bottom) row? Or is it enough to just have a set of consecutive values e.g. reading to to bottom the multicoloured array [6, 5, 8, 7, 4] would be a valid straight?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-04-26T18:15:10.413Z
Comments.Text: @NeilSlater Order does not matter. Updated. Thanks for pointing out.
Comments.UserDisplayName: durianice
Tags: <reinforcement-learning><deep-rl><reward-functions><reward-design><card-games>
Title: How should I write the reward function to teach the agent the rules of this card game?
UserDisplayName: durianice
ThreadText: # How should I write the reward function to teach the agent the rules of this card game?
  
  (Asked by: username_0 on 2022-04-26)
  
  I'm quite new to reinforcement learning. I've been training the model for the following problem but the mean reward is stuck.
  
  * In a 5 by 5 board, each position can contain a card with a color (0-4) and a value (0-9).
  * Some initial cards, all distinct from each other, may be on the board.
  * In each round, a card not on the board is chosen with uniform distribution.
  * The player has to place this card in an empty position before the next card is chosen.
  * When all positions are filled, the game ends and the final score is the sum of all points of each row, each column, and each of the two diagonals, where the points are calculated as follows (order does not matter, e.g. AAABB=ABABA, 12345=23415):
  
  | Combination | Points | Colors | Values |
  | --- | --- | --- | --- |
  | Straight flush | 30 | All identical | All consecutive |
  | Straight | 15 | Not all identical | All consecutive |
  | Flush | 14 | All identical | Not all consecutive |
  | Five of a kind | 28 |  | AAAAA |
  | Four of a kind | 16 |  | AAAAB |
  | Full house | 10 |  | AAABB |
  | Three of a kind | 6 |  | AAABC |
  | Two pairs | 3 |  | AABBC |
  | One pair | 1 |  | AABCD |
  | Nothing | 0 |  |  |
  
  The goal is of course to get a score as high as possible.
  
  Question:
  **How should I write the reward function to teach the agent the rules?**
  
  I used invalid action masking, so no invalid actions can be taken. However, it's still hard for high-score combinations to randomly show up. I tried setting reward to be the points possibly added due to the placed card (like if the placed card can form a pair with another one in the same row, add some points based on how much point a pair is rewarded in the final score), and reward 100\*(final score) at the end, hoping the agent to learn that the final score is the most important, but the mean reward was not improving even after 2 million timesteps.
  
  I use [MaskablePPO](https://sb3-contrib.readthedocs.io/en/master/modules/ppo_mask.html) with MlpPolicy (alias of MaskableActorCritic) from sb3-contrib. I represent each card as an int8. The upper 4 bits representing the color and the lower 4 bits represent the value.
  
  The observation space I use consists of the card to be placed, the cards on the board, and all cards not on the board. I'm wondering if I should include cards not present in the observation space. I tried to exclude it but it does not change the results much.
  
  Apart from direct answers, any pointers/reference would be much appreciated.
  
  **Tags:** reinforcement-learning, deep-rl, reward-functions, reward-design, card-games
  > **Comments:**
  > **username_1**: Is it a game that you can links the rules for? Your description is fine as-is, but the scoring rules may allow someone to try an experiment. One important detail: Is it possible to calculate a score for the incomplete board using the standard rules? As you have a large state space, and a policy network, I am assuming you are using a neural network. It may be worth explaining what you have tried there, and how you convert state to the representation used by the network. For instance, putting the raw 0-9 values in grid positions to feed into a CNN may not work well.  
  > **username_0**: @NeilSlater I updated some info. I am not using a CNN tho, because stable-baselines 3 only allows CNN to work with images. I'm using an actor critic policy.  
  > **username_1**: Thanks for the updates. Are the scores assigned according to strict sequence match ie. a straight has to have values in order in the horizontal (left to right) or vertical (top to bottom) row? Or is it enough to just have a set of consecutive values e.g. reading to to bottom the multicoloured array [6, 5, 8, 7, 4] would be a valid straight?  
  > **username_0**: @NeilSlater Order does not matter. Updated. Thanks for pointing out.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-04-26. Score: 1 | Accepted answer
  
  I would recommend having the reward as the increase to score caused by adding the card in the chosen location. You *could* optionally include calculations based on partial rows for pairs, three-of-a-kind etc. I do not think you should grant a different final reward for the end score, and definitely not some large difference, as the gradient for a large difference may swamp the networks (unless you also drop the learning rate, but then the smaller early values will not get learned from).
  
  I do not think that your focus should be on clever manipulation of reward signal. It is likely that looking at other things related to this problem will give you better impact.
  
  First you should look at your state representation. If you are feeding the `int8` values directly to a neural network, then this is very unlikely to work. Instead, you should use a short vector for each card you want to represent - of either 6 or 15 elements. You should be one-hot-encoding the colour for each position. You may find one-hot-encoding the value also helps, although that is trickier - if you don't one-hot encode, then you should scale the value e.g. $(v - 5)/3$ to keep it in range for best neural network learning.
  
  The card in hand, about to be placed, should be part of the state. So far, before any feature engineering, that would give you a vector of at least 26 \* 6 = 156 dimensions, but you could take that further.
  
  If possible, use [afterstates](https://stats.stackexchange.com/questions/411932/reinforcement-learning-afterstate-and-afterstate-value-functions) instead of enumerated actions. That may not be possible if you are using a library that expects enumerated actions. You probably won't find a version of Actor-Critic with afterstates because the Actor would need to express a probability distribution over the afterstates, which is awkward to do. You may be able to set the Actor to use enumerated actions and the Critic to score advantage based on afterstates though.
  
  To help the agent, do some feature engineering to add summaries of each possible line to the state (so a summary for each of the 12 lines). One important feature will be whether that line is complete or not - if it is complete, no more reward is possible from that line (perhaps a scaled "how many spaces left" counter would work here). Other features that could be useful are a "bag of colours" and "bag of values" i.e. one-hot-encoded list of current variation in that row - such a summary makes it easier for the NN to tell whether an x-of-a-kind or flush or straight are possible.
  
  In theory you could also pre-calculate the immediate reward from placing the card in each position, making that part of the state (another 25 features). That is so the agent is not trying to predict this value - it is hard for the NN, but easy for the game engine and not "cheating" at the game by e.g. peeking at the next card. That should allow the NN to use more of its capacity for planning ahead and statistically gambling on what might be available in future, which is perhaps the more interesting aspect of this game.
  
  
  ---
  
  