Answers.Body: <blockquote>
    <p>Would it be helpful to use a LSTM and reduce the input state?</p>
  </blockquote>
  
  <p>I'd bet, no. LSTM is more complicated and harder to learn, while the input is 4 * 9 * 36 bits is still rather limited.</p>
  
  <p>However, you may want to aggregate the information somehow, e.g., add additional bits informing about what cards were already played (no matter when). This information is redundant, but by providing it, you may save the network quite some learning.</p>
  
  <p>At the same time, you may want to use symmetries (all colors but trumps are equivalent and the therefore the weights should be the same).</p>
  
  <blockquote>
    <p>How to handle invalid moves?</p>
  </blockquote>
  
  <p>That's simple: There are no invalid moves. The network provides 36 outputs of how much it wants to play a given card. You simply take the one valid card having the greatest output value. You don't try to make the network learn what moves are valid as this is neither needed nor helpful.</p>
  
  <blockquote>
    <p>Do you have some good ideas for improvements? (like Neural-Fictitious Self-Play or something similar)</p>
  </blockquote>
  
  <p>I can't tell. But it should matter at the moment. First make you network clearly beat the random players, then you can look for more. Or start with Self-Play, as you want probably have both for comparison.</p>
  
  <blockquote>
    <p>Or is this the whole approach absolute nonsense?</p>
  </blockquote>
  
  <p>I don't think so, but ... (see below)</p>
  
  <blockquote>
    <p>I design the input state as a vector (one hot encoded) with 36 "bits" for every player</p>
  </blockquote>
  
  <p>This doesn't sound good. Every player has 9 of 36 cards and so should be the encoding. A player doesn't know the cards of other players.</p>
  
  <blockquote>
    <p>The reward is +1 for winning, -1 for losing,</p>
  </blockquote>
  
  <p>In most card games I know, it matters by how much you win (unlike e.g., in Go). Even when it doesn't matter, using this information at the early learning stages is IMHO useful.</p>
  
  <blockquote>
    <p>-0.1 for a invalid action and</p>
  </blockquote>
  
  <p>Drop the invalid action. Just transform anything the network produces to a valid action, add no penalty (as written above).</p>
  
  <blockquote>
    <p>... 0 for an action which doesn't lead to a terminal state</p>
  </blockquote>
  
  <p>All action but the last one lead to a non-terminal state. You can use some Temporal Difference Learning or use the fact that the game has a small fixed number of moves and reward/punish all actions takes in a the whole game.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-03T18:59:36.047Z
Answers.Text: Thank you very much for your detailed answer. I will try your hints a let you know if I have some achievements.
Answers.UserDisplayName: murthy10
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-02T20:22:46.973Z
Answers.LastActivityDate: 2018-04-02T20:22:46.973Z
Answers.UserDisplayName: maaartinus
Body: <p>I'm currently having troubles to win against a random bot playing the Schieber Jass game. It is a imperfect card information game. (famous in switzerland <a href="https://www.schieber.ch/" rel="nofollow noreferrer">https://www.schieber.ch/</a>) </p>
  
  <p>The environement I'm using is on Github <a href="https://github.com/murthy10/pyschieber" rel="nofollow noreferrer">https://github.com/murthy10/pyschieber</a></p>
  
  <p>To get a brief overview of the Schieber Jass I will describe the main characteristics of the game.
  The Schieber Jass consists of four players building two teams.
  At the beginning every player gets randomly nine cards (there are 36 cards).
  Now there are nine rounds and every player has to chose one card every round. Related to the rules of the game the "highest card" wins and the team gets the points.
  Hence the goal is to get more points then your opponent team.</p>
  
  <p>There are several more rules but I think you can image how the game should roughly work.</p>
  
  <p>Now I'm trying to apply a DQN approach at the game.</p>
  
  <p>To my attempts:</p>
  
  <ul>
  <li>I let two independent reinforcement player play against two random players</li>
  <li>I design the input state as a vector (one hot encoded) with 36 "bits" for every player and repeated this nine times for every card you can play during a game.</li>
  <li>The output is a vector of 36 "bits" for every possible card.</li>
  <li>If the greedy output of the network suggest an invalid action I take the action with the highest probability of the allowed actions</li>
  <li>The reward is +1 for winning, -1 for losing, -0.1 for a invalid action and 0 for an action which doesn't lead to a terminal state</li>
  </ul>
  
  <p>My question:</p>
  
  <ul>
  <li>Would it be helpful to use a LSTM and reduce the input state?</li>
  <li>How to handle invalid moves?</li>
  <li>Do you have some good ideas for improvements? (like Neural-Fictitious Self-Play or something similar)</li>
  <li>Or is this the whole approach absolute nonsense?</li>
  </ul>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2018-03-28T18:43:25.057Z
Comments.Text: Welcome to AI!  Sounds like an interesting project.
Comments.UserDisplayName: DukeZhou
Tags: <reinforcement-learning><game-ai><q-learning><imperfect-information>
Title: How to use DQN to handle an imperfect but complete information game?
UserDisplayName: murthy10
ThreadText: # How to use DQN to handle an imperfect but complete information game?
  
  (Asked by: username_0 on 2018-03-27)
  
  I'm currently having troubles to win against a random bot playing the Schieber Jass game. It is a imperfect card information game. (famous in switzerland <https://www.schieber.ch/>)
  
  The environement I'm using is on Github <https://github.com/murthy10/pyschieber>
  
  To get a brief overview of the Schieber Jass I will describe the main characteristics of the game.
  The Schieber Jass consists of four players building two teams.
  At the beginning every player gets randomly nine cards (there are 36 cards).
  Now there are nine rounds and every player has to chose one card every round. Related to the rules of the game the "highest card" wins and the team gets the points.
  Hence the goal is to get more points then your opponent team.
  
  There are several more rules but I think you can image how the game should roughly work.
  
  Now I'm trying to apply a DQN approach at the game.
  
  To my attempts:
  
  * I let two independent reinforcement player play against two random players
  * I design the input state as a vector (one hot encoded) with 36 "bits" for every player and repeated this nine times for every card you can play during a game.
  * The output is a vector of 36 "bits" for every possible card.
  * If the greedy output of the network suggest an invalid action I take the action with the highest probability of the allowed actions
  * The reward is +1 for winning, -1 for losing, -0.1 for a invalid action and 0 for an action which doesn't lead to a terminal state
  
  My question:
  
  * Would it be helpful to use a LSTM and reduce the input state?
  * How to handle invalid moves?
  * Do you have some good ideas for improvements? (like Neural-Fictitious Self-Play or something similar)
  * Or is this the whole approach absolute nonsense?
  
  **Tags:** reinforcement-learning, game-ai, q-learning, imperfect-information
  > **Comments:**
  > **username_1**: Welcome to AI!  Sounds like an interesting project.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2018-04-02. Score: 3
  
  > Would it be helpful to use a LSTM and reduce the input state?
  
  I'd bet, no. LSTM is more complicated and harder to learn, while the input is 4 \* 9 \* 36 bits is still rather limited.
  
  However, you may want to aggregate the information somehow, e.g., add additional bits informing about what cards were already played (no matter when). This information is redundant, but by providing it, you may save the network quite some learning.
  
  At the same time, you may want to use symmetries (all colors but trumps are equivalent and the therefore the weights should be the same).
  
  > How to handle invalid moves?
  
  That's simple: There are no invalid moves. The network provides 36 outputs of how much it wants to play a given card. You simply take the one valid card having the greatest output value. You don't try to make the network learn what moves are valid as this is neither needed nor helpful.
  
  > Do you have some good ideas for improvements? (like Neural-Fictitious Self-Play or something similar)
  
  I can't tell. But it should matter at the moment. First make you network clearly beat the random players, then you can look for more. Or start with Self-Play, as you want probably have both for comparison.
  
  > Or is this the whole approach absolute nonsense?
  
  I don't think so, but ... (see below)
  
  > I design the input state as a vector (one hot encoded) with 36 "bits" for every player
  
  This doesn't sound good. Every player has 9 of 36 cards and so should be the encoding. A player doesn't know the cards of other players.
  
  > The reward is +1 for winning, -1 for losing,
  
  In most card games I know, it matters by how much you win (unlike e.g., in Go). Even when it doesn't matter, using this information at the early learning stages is IMHO useful.
  
  > -0.1 for a invalid action and
  
  Drop the invalid action. Just transform anything the network produces to a valid action, add no penalty (as written above).
  
  > ... 0 for an action which doesn't lead to a terminal state
  
  All action but the last one lead to a non-terminal state. You can use some Temporal Difference Learning or use the fact that the game has a small fixed number of moves and reward/punish all actions takes in a the whole game.
  
  > **Comments:**
  > **username_0**: Thank you very much for your detailed answer. I will try your hints a let you know if I have some achievements.  
  
  ---
  
  