Answers.Body: <p>I don't think your network, trained using PPO to play a card game, already contains sufficient information to also use for drafting. I'm not saying this with 100% certainty, maybe there's something I'm overlooking, but I can't think of anything right now.</p>
  
  <p>A small adaptation to the network might be sufficient (though it would also involve re-training again). Recently, OpenAI has been writing about their attempts to train agents to play the game DOTA 2. Now, this isn't a card game, it doesn't require deckbuilding, but there is an aspect to the game that is somewhat similar to deckbuilding: <em>drafting</em>. In DOTA 2, there are two teams of 5 players each. Before a game start, each team selects 5 <em>heroes</em> (one per player) to play in that game. This is very similar to deckbuilding, except that it's likely a much smaller problem; there's only a "deck" (team composition) of 5 "cards" (heroes).</p>
  
  <p>Anyway, they also trained agents to play the game (controlling one hero per agent) using PPO. In a <a href="https://blog.openai.com/openai-five-benchmark-results/" rel="nofollow noreferrer">blog post</a>, they write the following about how they managed to add drafting capabilities relatively easily:</p>
  
  <blockquote>
    <p>In late June we added a win probability output to our neural network to introspect what OpenAI Five is predicting. When later considering drafting, we realized we could use this to evaluate the win probability of any draft: just look at the prediction on the first frame of a game with that lineup. In one week of implementation, we crafted a fake frame for each of the 11 million possible team matchups and wrote a tree search to find OpenAI Five’s optimal draft.</p>
  </blockquote>
  
  <p>So, if you want to try a similar technique, you'd have to adapt your network such that it also learns to generate a prediction of the win probability as output. I imagine that it'd be much less effective for deckbuilding, because win probabilities may all be very close to 50% in card games where luck (when drawing cards for example) can be a significant factor, but it might be worth a try.</p>
  
  <hr>
  
  <p>Alternatively, instead of generating lots of random decks and playing with them all, you could view the problem of deckbuilding as an additional separate "game" or Markov Decision Process; adding a specific card to the deck can be an action, and this MDP terminates once you have a complete deck. Then you can try to do that better than random using search algorithms (like Monte-Carlo Tree Search) or, again, a Reinforcement Learning approach like PPO. Again, I imagine it will be a very difficult problem though, likely requiring lots of time before it will be capable of doing better than random.</p>
  
  <hr>
  
  <p>I also know of some research related to deckbuilding in the collectible card game <em>Hearthstone</em>, which may be relevant for you. Unfortunately I did not yet get to read through any of this in detail, so I don't know for sure if you'll find a solution here, but it may be worth a try:</p>
  
  <ul>
  <li><a href="https://dockhorn.antares.uberspace.de/wordpress/additional-material/" rel="nofollow noreferrer">https://dockhorn.antares.uberspace.de/wordpress/additional-material/</a>: At the recent IEEE CIG 2018 conference, there were two competitions involving Hearthstone: one for game-playing AI with pre-built decks, and for generating new decks (and then playing with them). The page I linked to here contains references to related work from earlier years.</li>
  <li><a href="https://arxiv.org/abs/1806.09771" rel="nofollow noreferrer">Q-DeckRec: A Fast Deck Recommendation System for Collectible Card Games</a>: This is a new paper that was published in the proceedings of this recent IEEE CIG 2018 conference.</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-21T15:52:24.657Z
Answers.LastActivityDate: 2018-08-21T15:52:24.657Z
Answers.UserDisplayName: Dennis Soemers
Body: <p>I've developed a neural network that can play a card game.  I now want to use it to create decks for the game.  My first thought would be to run a lot of games with random decks and use some approximation (maybe just a linear approximation with a feature for each card in your hand) to learn the value function for each state.  </p>
  
  <p>However, this will probably take a while, so in the mean time is there any way I could get this information directly from the neural network? </p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-08-20T18:17:50.127Z
Comments.Text: What kind of neural network do you have / how did you train it? What outputs does it have (did you train a network to output values with an algorithm like DQN, or did you train it to output a policy using policy gradient methods, or something else?)
Comments.UserDisplayName: Dennis Soemers
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-08-20T18:33:28.960Z
Comments.Text: This may or may not be relevant, but I'm personally interested in what type of card game it is.  *Deck building implies some form of post-Garfield [CCG](https://en.wikipedia.org/wiki/Collectible_card_game).  Algorithmic deck building is an exciting prospect!*  Can I ask, how many cards in the card "library" which a deck can be built from?
Comments.UserDisplayName: DukeZhou
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-08-21T10:09:37.710Z
Comments.Text: @DennisSoemers There is 1 output for every action that could ever be played (almost all of this is playing card X from your hand), the highest output that is a valid move is selected to play.  I used PPO to train the network.
Comments.UserDisplayName: OrangeMan
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-08-21T10:10:36.727Z
Comments.Text: @DukeZhou There are about 500 cards in the library right now.
Comments.UserDisplayName: OrangeMan
Tags: <neural-networks><game-ai>
Title: Can you analyse a neural network to determine good states?
UserDisplayName: OrangeMan
ThreadText: # Can you analyse a neural network to determine good states?
  
  (Asked by: username_0 on 2018-08-20)
  
  I've developed a neural network that can play a card game. I now want to use it to create decks for the game. My first thought would be to run a lot of games with random decks and use some approximation (maybe just a linear approximation with a feature for each card in your hand) to learn the value function for each state.
  
  However, this will probably take a while, so in the mean time is there any way I could get this information directly from the neural network?
  
  **Tags:** neural-networks, game-ai
  
  > **Comments:**
  > **username_1**: What kind of neural network do you have / how did you train it? What outputs does it have (did you train a network to output values with an algorithm like DQN, or did you train it to output a policy using policy gradient methods, or something else?)  
  > **username_2**: This may or may not be relevant, but I'm personally interested in what type of card game it is.  *Deck building implies some form of post-Garfield [CCG](https://en.wikipedia.org/wiki/Collectible_card_game).  Algorithmic deck building is an exciting prospect!*  Can I ask, how many cards in the card "library" which a deck can be built from?  
  > **username_0**: @username_1 There is 1 output for every action that could ever be played (almost all of this is playing card X from your hand), the highest output that is a valid move is selected to play.  I used PPO to train the network.  
  > **username_0**: @username_2 There are about 500 cards in the library right now.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-08-21. Score: 4 | Accepted answer
  
  I don't think your network, trained using PPO to play a card game, already contains sufficient information to also use for drafting. I'm not saying this with 100% certainty, maybe there's something I'm overlooking, but I can't think of anything right now.
  
  A small adaptation to the network might be sufficient (though it would also involve re-training again). Recently, OpenAI has been writing about their attempts to train agents to play the game DOTA 2. Now, this isn't a card game, it doesn't require deckbuilding, but there is an aspect to the game that is somewhat similar to deckbuilding: *drafting*. In DOTA 2, there are two teams of 5 players each. Before a game start, each team selects 5 *heroes* (one per player) to play in that game. This is very similar to deckbuilding, except that it's likely a much smaller problem; there's only a "deck" (team composition) of 5 "cards" (heroes).
  
  Anyway, they also trained agents to play the game (controlling one hero per agent) using PPO. In a [blog post](https://blog.openai.com/openai-five-benchmark-results/), they write the following about how they managed to add drafting capabilities relatively easily:
  
  > In late June we added a win probability output to our neural network to introspect what OpenAI Five is predicting. When later considering drafting, we realized we could use this to evaluate the win probability of any draft: just look at the prediction on the first frame of a game with that lineup. In one week of implementation, we crafted a fake frame for each of the 11 million possible team matchups and wrote a tree search to find OpenAI Five’s optimal draft.
  
  So, if you want to try a similar technique, you'd have to adapt your network such that it also learns to generate a prediction of the win probability as output. I imagine that it'd be much less effective for deckbuilding, because win probabilities may all be very close to 50% in card games where luck (when drawing cards for example) can be a significant factor, but it might be worth a try.
  
  ---
  
  Alternatively, instead of generating lots of random decks and playing with them all, you could view the problem of deckbuilding as an additional separate "game" or Markov Decision Process; adding a specific card to the deck can be an action, and this MDP terminates once you have a complete deck. Then you can try to do that better than random using search algorithms (like Monte-Carlo Tree Search) or, again, a Reinforcement Learning approach like PPO. Again, I imagine it will be a very difficult problem though, likely requiring lots of time before it will be capable of doing better than random.
  
  ---
  
  I also know of some research related to deckbuilding in the collectible card game *Hearthstone*, which may be relevant for you. Unfortunately I did not yet get to read through any of this in detail, so I don't know for sure if you'll find a solution here, but it may be worth a try:
  
  * <https://dockhorn.antares.uberspace.de/wordpress/additional-material/>: At the recent IEEE CIG 2018 conference, there were two competitions involving Hearthstone: one for game-playing AI with pre-built decks, and for generating new decks (and then playing with them). The page I linked to here contains references to related work from earlier years.
  * [Q-DeckRec: A Fast Deck Recommendation System for Collectible Card Games](https://arxiv.org/abs/1806.09771): This is a new paper that was published in the proceedings of this recent IEEE CIG 2018 conference.
  
  
  ---
  
  