Answers.Body: <p>Because your action space is not discrete, It would not be wise to evaluate possible actions. What would probably be better is evaluating next states. This means evaluating the value of the state after the action with your neural network instead of the value of a action. </p>
  
  <p>Why is this better? 
  You can use the neural net for each possible action , instead of trying to make the neural network figure out which actions are possible. the result would be taking the action of which the next state has the most value.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-09T19:36:55.697Z
Answers.Text: Could you point me to some examples?
Answers.UserDisplayName: sqp_125
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-10T13:22:27.863Z
Answers.Text: sure, I found this answer on this website : https://ai.stackexchange.com/questions/9491/inconsistent-action-space-in-reinforcement-learning
Answers.UserDisplayName: Lustwelpintje
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-10T15:37:18.603Z
Answers.Text: I just read this article but I cannot find what you are suggesting. What I could directly re-implement is choice (1): Such that the network learns possible actions but this is as said not the best option. Do you know some code example / what would be the name of this algorithm you suggested  (I think it is not DQN anymore). Sry I am completely new to in this topic.
Answers.UserDisplayName: sqp_125
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-11T09:03:47.960Z
Answers.Text: Try looking up on state value functions vs state action value functions. I remember learning this from the book 'Reinforcement learning : an introduction'. from A quick internet search maybe this will help you understand: https://www.cs.dartmouth.edu/~lorenzo/teaching/cs134/Archive/Spring2009/final/PengTao/final_report.pdf
Answers.UserDisplayName: Lustwelpintje
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-09T10:57:00.760Z
Answers.LastActivityDate: 2019-12-09T10:57:00.760Z
Answers.UserDisplayName: Lustwelpintje
Body: <p>I want to train a DQN card game named witches. It consists of 60 Cards (1-14 of Yellow, Blue, Green, Red Cards + 4 Wizzards). The color of the first layed card has to be respected by the other players (if they have this card in hand). The one who has the card with the highest number gets the played cards. Each collected red card gives you a -1 point. </p>
  
  <p>With respect to this <a href="https://ai.stackexchange.com/questions/7049/dqn-input-representation-for-a-card-game/7050?noredirect=1#comment25924_7050">answer</a> I setup the inputs / state of the NN as binary (meaning I have 180 bool values (60 card x is currently on the table, 60 card x is in the ai hand, 60 card x is already been played)</p>
  
  <p><strong>How to design the outputs / actions?</strong></p>
  
  <ul>
  <li>If the ai is the first player of a round it can play any card</li>
  <li>If the ai is not first player it has to respect the first played card (or play a wizzard)</li>
  </ul>
  
  <p>This means there is actually a list of available options. I then sort this list and have 60 Output bools which I set to 1 if this option is possible. Among these options the ai should then decide what the correct option is? Is this the correct procedure?</p>
  
  <p><strong>Inconsistent / Varying Action Space</strong>
  This is what we have to deal here with. As explained in <a href="https://ai.stackexchange.com/questions/9491/inconsistent-action-space-in-reinforcement-learning">here</a> I think a DQN as well as Policy Gradient Methods is not the correct architecture to choose for solving such multi-agent card games. What architecture would you choose?</p>
  
  <p><strong>General procedure?</strong></p>
  
  <p>Assume I have 4 players, so do I have to get the old state before the ai is the next player and the new state is directly after this round is finished?</p>
  
  <pre><code>my_game = game(["Tim", "Bob", "Lena", "Anja"])
  while True:
      #1. Play unti AI has to move:
      my_game.play_round_until_ai()
  
      #2. Get old state:
      state_old = agent.get_state(my_game)
  
      #3. Get the action the AI should perform
      action_ = agent.get_action(state_old, my_game)
  
      #4. perform new Action and get new state
      #reward rates how good the current action was
      #score is the actual score of this game!
      reward, done, score = my_game.finishRound(action_)
  
      # 5: Calculate new state
      state_new = agent.get_state(my_game)
  
      #6. train short memory base on the new action and state
      agent.train_short_memory(state_old, action_, reward, state_new, done)
  
      #7. store the new data into a long term memory
      agent.remember(state_old, action_, reward, state_new, done)
  
      if done == True:
          # One game is over, train on the memory and plot the result.
          sc = my_game.reset()
  </code></pre>
  
  <p>My code so far is available here: <a href="https://github.com/CesMak/witches_ai" rel="nofollow noreferrer">https://github.com/CesMak/witches_ai</a> </p>
  
Tags: <dqn>
Title: DQN card game - how to represent the actions?
UserDisplayName: sqp_125
ThreadText: # DQN card game - how to represent the actions?
  
  (Asked by: username_0 on 2019-12-09)
  
  I want to train a DQN card game named witches. It consists of 60 Cards (1-14 of Yellow, Blue, Green, Red Cards + 4 Wizzards). The color of the first layed card has to be respected by the other players (if they have this card in hand). The one who has the card with the highest number gets the played cards. Each collected red card gives you a -1 point. 
  With respect to this answer I setup the inputs / state of the NN as binary (meaning I have 180 bool values (60 card x is currently on the table, 60 card x is in the ai hand, 60 card x is already been played)
  How to design the outputs / actions?
  
  If the ai is the first player of a round it can play any card
  If the ai is not first player it has to respect the first played card (or play a wizzard)
  
  This means there is actually a list of available options. I then sort this list and have 60 Output bools which I set to 1 if this option is possible. Among these options the ai should then decide what the correct option is? Is this the correct procedure?
  Inconsistent / Varying Action Space
  This is what we have to deal here with. As explained in here I think a DQN as well as Policy Gradient Methods is not the correct architecture to choose for solving such multi-agent card games. What architecture would you choose?
  General procedure?
  Assume I have 4 players, so do I have to get the old state before the ai is the next player and the new state is directly after this round is finished?
  my_game = game(["Tim", "Bob", "Lena", "Anja"])
  while True:
      #1. Play unti AI has to move:
      my_game.play_round_until_ai()
  
      #2. Get old state:
      state_old = agent.get_state(my_game)
  
      #3. Get the action the AI should perform
      action_ = agent.get_action(state_old, my_game)
  
      #4. perform new Action and get new state
      #reward rates how good the current action was
      #score is the actual score of this game!
      reward, done, score = my_game.finishRound(action_)
  
      # 5: Calculate new state
      state_new = agent.get_state(my_game)
  
      #6. train short memory base on the new action and state
      agent.train_short_memory(state_old, action_, reward, state_new, done)
  
      #7. store the new data into a long term memory
      agent.remember(state_old, action_, reward, state_new, done)
  
      if done == True:
          # One game is over, train on the memory and plot the result.
          sc = my_game.reset()
  
  My code so far is available here: https://github.com/CesMak/witches_ai
  
  **Tags:** dqn
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-12-09. Score: 1
  
  Because your action space is not discrete, It would not be wise to evaluate possible actions. What would probably be better is evaluating next states. This means evaluating the value of the state after the action with your neural network instead of the value of a action.
  
  Why is this better?
  You can use the neural net for each possible action , instead of trying to make the neural network figure out which actions are possible. the result would be taking the action of which the next state has the most value.
  
  > **Comments:**
  > **username_0**: Could you point me to some examples?  
  > **username_1**: sure, I found this answer on this website : https://ai.stackexchange.com/questions/9491/inconsistent-action-space-in-reinforcement-learning  
  > **username_0**: I just read this article but I cannot find what you are suggesting. What I could directly re-implement is choice (1): Such that the network learns possible actions but this is as said not the best option. Do you know some code example / what would be the name of this algorithm you suggested  (I think it is not DQN anymore). Sry I am completely new to in this topic.  
  > **username_1**: Try looking up on state value functions vs state action value functions. I remember learning this from the book 'Reinforcement learning : an introduction'. from A quick internet search maybe this will help you understand: https://www.cs.dartmouth.edu/~lorenzo/teaching/cs134/Archive/Spring2009/final/PengTao/final_report.pdf  
  
  ---
  
  