Episodic Exploration for Deep Deterministic Policies for StarCraft Micromanagement
We consider scenarios from the real-time strategy game StarCraft as benchmarks for reinforcement learning algorithms. We focus on micromanagement, that is, the short-term, low-level control of team members during a battle. We propose several scenarios that are challenging for reinforcement learning algorithms because the state- action space is very large, and there is no obvious feature representation for the value functions. We describe our approach to tackle the micromanagement scenarios with deep neural network controllers from raw state features given by the game engine. We also present a heuristic reinforcement learning algorithm which combines direct exploration in the policy space and backpropagation. This algorithm collects traces for learning using deterministic policies, which appears much more efficient than, e.g., ?-greedy exploration. Experiments show that this algorithm allows to successfully learn non-trivial strategies for scenarios with armies of up to 15 agents, where both Q-learning and REINFORCE struggle.
This work introduces some StarCraft micro-management tasks (controlling individual units during a battle). These tasks are difficult for recent DeepRL methods due to high-dimensional, variable action spaces (the action space is the task of each unit, the number of units may vary). In such large action spaces, simple exploration strategies (such as epsilon-greedy) perform poorly. They introduce a novel algorithm ZO to tackle this problem. This algorithm combines ideas from policy gradient, deep networks trained with backpropagation for state embedding and gradient free optimization. The algorithm is well explained and is compared to some existing baselines. Due to the gradient free optimization providing for much better structured exploration, it performs far better. This is a well-written paper and a novel algorithm which is applied to a very relevant problem. After the success of DeepRL approaches at learning in large state spaces such as visual environment, there is significant interest in applying RL to more structured state and action spaces. The tasks introduced here are interesting environments for these sorts of problems. It would be helpful if the authors were able to share the source code / specifications for their tasks, to allow other groups to compare against this work. I found section 5 (the details of the raw inputs and feature encodings) somewhat difficult to understand. In addition to clarifying, the authors might wish to consider whether they could provide the source code to their algorithm or at least the encoder to allow careful comparisons by other work. Although discussed, there is no baseline comparison with valued based approaches with attempt to do better exploration by modeling uncertainty (such as Bootstrapped DQN). It would useful to understand how such approaches, which also promise better exploration, compare. It would also be interesting to discuss whether action embedding models such as energy-based approaches (e.g. https://arxiv.org/pdf/1512.07679v2.pdf, http://www.jmlr.org/papers/volume5/sallans04a/sallans04a.pdf ) or continuous action embeddings (https://arxiv.org/pdf/1512.07679v2.pdf ) would provide an alternative approach for structured exploration in these action spaces.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Episodic Exploration for Deep Deterministic Policies for StarCraft Micromanagement
We consider scenarios from the real-time strategy game StarCraft as benchmarks for reinforcement learning algorithms. We focus on micromanagement, that is, the short-term, low-level control of team members during a battle. We propose several scenarios that are challenging for reinforcement learning algorithms because the state- action space is very large, and there is no obvious feature representation for the value functions. We describe our approach to tackle the micromanagement scenarios with deep neural network controllers from raw state features given by the game engine. We also present a heuristic reinforcement learning algorithm which combines direct exploration in the policy space and backpropagation. This algorithm collects traces for learning using deterministic policies, which appears much more efficient than, e.g., ?-greedy exploration. Experiments show that this algorithm allows to successfully learn non-trivial strategies for scenarios with armies of up to 15 agents, where both Q-learning and REINFORCE struggle.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



This work introduces some StarCraft micro-management tasks (controlling individual units during a battle). These tasks are difficult for recent DeepRL methods due to high-dimensional, variable action spaces (the action space is the task of each unit, the number of units may vary). In such large action spaces, simple exploration strategies (such as epsilon-greedy) perform poorly. They introduce a novel algorithm ZO to tackle this problem. This algorithm combines ideas from policy gradient, deep networks trained with backpropagation for state embedding and gradient free optimization. The algorithm is well explained and is compared to some existing baselines. Due to the gradient free optimization providing for much better structured exploration, it performs far better. This is a well-written paper and a novel algorithm which is applied to a very relevant problem. After the success of DeepRL approaches at learning in large state spaces such as visual environment, there is significant interest in applying RL to more structured state and action spaces. The tasks introduced here are interesting environments for these sorts of problems. It would be helpful if the authors were able to share the source code / specifications for their tasks, to allow other groups to compare against this work. I found section 5 (the details of the raw inputs and feature encodings) somewhat difficult to understand. In addition to clarifying, the authors might wish to consider whether they could provide the source code to their algorithm or at least the encoder to allow careful comparisons by other work. Although discussed, there is no baseline comparison with valued based approaches with attempt to do better exploration by modeling uncertainty (such as Bootstrapped DQN). It would useful to understand how such approaches, which also promise better exploration, compare. It would also be interesting to discuss whether action embedding models such as energy-based approaches (e.g. https://arxiv.org/pdf/1512.07679v2.pdf, http://www.jmlr.org/papers/volume5/sallans04a/sallans04a.pdf ) or continuous action embeddings (https://arxiv.org/pdf/1512.07679v2.pdf ) would provide an alternative approach for structured exploration in these action spaces.
assistant
This work introduces some StarCraft micro-management tasks (controlling individual units during a battle). These tasks are difficult for recent DeepRL methods due to high-dimensional, variable action spaces (the action space is the task of each unit, the number of units may vary). In such large action spaces, simple exploration strategies (such as epsilon-greedy) perform poorly. They introduce a novel algorithm ZO to tackle this problem. This algorithm combines ideas from policy gradient, deep networks trained with backpropagation for state embedding and gradient free optimization. The algorithm is well explained and is compared to some existing baselines. Due to the gradient free optimization providing for much better structured exploration, it performs far better. This is a well-written paper and a novel algorithm which is applied to a very relevant problem. After the success of DeepRL approaches at learning in large state spaces such as visual environment, there is significant interest in applying RL to more structured state and action spaces. The tasks introduced here are interesting environments for these sorts of problems. It would be helpful if the authors were able to share the source code / specifications for their tasks, to allow other groups to compare against this work. I found section 5 (the details of the raw inputs and feature encodings) somewhat difficult to understand. In addition to clarifying, the authors might wish to consider whether they could provide the source code to their algorithm or at least the encoder to allow careful comparisons by other work. Although discussed, there is no baseline comparison with valued based approaches with attempt to do better exploration by modeling uncertainty (such as Bootstrapped DQN). It would useful to understand how such approaches, which also promise better exploration, compare. It would also be interesting to discuss whether action embedding models such as energy-based approaches (e.g. https://arxiv.org/pdf/1512.07679v2.pdf, http://www.jmlr.org/papers/volume5/sallans04a/sallans04a.pdf ) or continuous action embeddings (https://arxiv.org/pdf/1512.07679v2.pdf ) would provide an alternative approach for structured exploration in these action spaces.