Answers.Body: <p>The master's thesis <a href="https://project.dke.maastrichtuniversity.nl/games/files/msc/Roelofs_thesis.pdf" rel="nofollow noreferrer">Action space representation in combinatorial multi-armed bandits</a> (2015) seems to provide an answer to my question.</p>
  <p>Several algorithms can be used</p>
  <ul>
  <li>Naive Monte-Carlo Sampling (NMC)</li>
  <li>Linear Side Information (LSI)</li>
  <li>Monte Carlo Tree Search with Hierarchical Expansion (MCTS-HE)</li>
  <li>MCTS with Dimensional Expansion</li>
  </ul>
  <p>The idea is to divide and conquer.</p>
  <p>Here's a screenshot of the HE algorithm from <a href="https://project.dke.maastrichtuniversity.nl/games/files/msc/Roelofs_thesis.pdf#page=32" rel="nofollow noreferrer">section 4.2</a>.</p>
  <p><a href="https://i.sstatic.net/X7ABX.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/X7ABX.png" alt="enter image description here" /></a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-12-24T20:47:49.907Z
Answers.LastActivityDate: 2021-01-17T15:13:40.017Z
Answers.LastEditDate: 2021-01-17T15:13:40.017Z
Answers.UserDisplayName: Carrier Battles
Answers.Body: <p>One very popular RL algorithm that is capable of predicting multiple action outputs concurrently is <a href="https://arxiv.org/abs/1707.06347" rel="nofollow noreferrer">Proximal Policy Optimization</a>. In that algorithm, one or more, say <span class="math-container">$n$</span>, tuples of outputs, <span class="math-container">$(\mu, \sigma)$</span>, can be predicted at once (having <span class="math-container">$2*n$</span> output nodes), where each tuple is used to parameterize a Gaussian distribution from which a respective action value is sampled. The thus sampled action values are then applied in the simulation/game. By slightly modifying this procedure, this can of course also be applied to discrete action spaces equally well.</p>
  <p>To get you started, multiple high-quality implementations of PPO are available for rapid prototyping, e.g. <a href="https://github.com/openai/baselines" rel="nofollow noreferrer">OpenAI baselines</a> or <a href="https://stable-baselines.readthedocs.io/en/master/" rel="nofollow noreferrer">stable-baselines</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-14T21:52:19.103Z
Answers.LastActivityDate: 2021-01-14T21:52:19.103Z
Answers.UserDisplayName: Daniel B.
Body: <p>I have a steady hex-map and turn-based wargame featuring WWII carrier battles.</p>
  <p>On a given turn, a player may choose to perform a large number of actions. Actions can be of many different types, and some actions may be performed independently of each other while others have dependencies. For example, a player may decide to move one or two naval units, then assign a mission to an air unit or not, then adjust some battle parameters or not, and then reorganize a naval task force or not.</p>
  <p> Usually, boardgames allow players to perform only one action each turn (e.g. go or chess) or a few very similar actions (backgammon).</p>
  <p>Here the player may select</p>
  <ul>
  <li>Several actions</li>
  <li>The actions are of different nature</li>
  <li>Each action may have parameters that the player must set (e.g. strength, payload, destination)</li>
  </ul>
  <p><strong>How could I approach this problem with reinforcement learning? How would I specify a model or train it effectively to play such a game?</strong></p>
  <p>Here is a screenshot of the game.</p>
  <p><a href="https://i.sstatic.net/21wjd.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/21wjd.png" alt="enter image description here" /></a></p>
  <p>Here's another.</p>
  <p><a href="https://i.sstatic.net/vgUiV.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/vgUiV.png" alt="enter image description here" /></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-17T15:15:07.277Z
Comments.Text: This an old question, but could you please provide the name of this game? Moreover, how are you actually interacting with the game? Is there an available public API that the game developers provide so that users can program their own AI? Maybe this is a game you're developing. It seems that the game is this one: [https://apprecs.com/ios/961994680/carrier-battles-4-guadalcanal](https://apprecs.com/ios/961994680/carrier-battles-4-guadalcanal) or https://carrier-battles.com/.
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><game-ai><monte-carlo-tree-search><action-spaces><hierarchical-rl>
Title: How can I design a reinforcement learning model for a game with multiple complex actions taken at a time?
UserDisplayName: Carrier Battles
ThreadText: # How can I design a reinforcement learning model for a game with multiple complex actions taken at a time?
  
  (Asked by: username_0 on 2018-12-23)
  
  I have a steady hex-map and turn-based wargame featuring WWII username_0 username_0.
  
  On a given turn, a player may choose to perform a large number of actions. Actions can be of many different types, and some actions may be performed independently of each other while others have dependencies. For example, a player may decide to move one or two naval units, then assign a mission to an air unit or not, then adjust some battle parameters or not, and then reorganize a naval task force or not.
  
   Usually, boardgames allow players to perform only one action each turn (e.g. go or chess) or a few very similar actions (backgammon).
  
  Here the player may select
  
  * Several actions
  * The actions are of different nature
  * Each action may have parameters that the player must set (e.g. strength, payload, destination)
  
  **How could I approach this problem with reinforcement learning? How would I specify a model or train it effectively to play such a game?**
  
  Here is a screenshot of the game.
  
  [![enter image description here](https://i.sstatic.net/21wjd.png)](https://i.sstatic.net/21wjd.png)
  
  Here's another.
  
  [![enter image description here](https://i.sstatic.net/vgUiV.png)](https://i.sstatic.net/vgUiV.png)
  
  **Tags:** reinforcement-learning, game-ai, monte-carlo-tree-search, action-spaces, hierarchical-rl
  
  > **Comments:**
  > **username_1**: This an old question, but could you please provide the name of this game? Moreover, how are you actually interacting with the game? Is there an available public API that the game developers provide so that users can program their own AI? Maybe this is a game you're developing. It seems that the game is this one: [https://apprecs.com/ios/961994680/username_0-username_0-4-guadalcanal](https://apprecs.com/ios/961994680/username_0-username_0-4-guadalcanal) or https://username_0-username_0.com/.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_0 on 2018-12-24. Score: 1
  
  The master's thesis [Action space representation in combinatorial multi-armed bandits](https://project.dke.maastrichtuniversity.nl/games/files/msc/Roelofs_thesis.pdf) (2015) seems to provide an answer to my question.
  
  Several algorithms can be used
  
  * Naive Monte-Carlo Sampling (NMC)
  * Linear Side Information (LSI)
  * Monte Carlo Tree Search with Hierarchical Expansion (MCTS-HE)
  * MCTS with Dimensional Expansion
  
  The idea is to divide and conquer.
  
  Here's a screenshot of the HE algorithm from [section 4.2](https://project.dke.maastrichtuniversity.nl/games/files/msc/Roelofs_thesis.pdf#page=32).
  
  [![enter image description here](https://i.sstatic.net/X7ABX.png)](https://i.sstatic.net/X7ABX.png)
  
  
  ---
  
  ### Answer by username_2 on 2021-01-14. Score: 0
  
  One very popular RL algorithm that is capable of predicting multiple action outputs concurrently is [Proximal Policy Optimization](https://arxiv.org/abs/1707.06347). In that algorithm, one or more, say $n$, tuples of outputs, $(\mu, \sigma)$, can be predicted at once (having $2\*n$ output nodes), where each tuple is used to parameterize a Gaussian distribution from which a respective action value is sampled. The thus sampled action values are then applied in the simulation/game. By slightly modifying this procedure, this can of course also be applied to discrete action spaces equally well.
  
  To get you started, multiple high-quality implementations of PPO are available for rapid prototyping, e.g. [OpenAI baselines](https://github.com/openai/baselines) or [stable-baselines](https://stable-baselines.readthedocs.io/en/master/).
  
  
  ---
  
  