PARAMETRIZED DEEP Q-NETWORKS LEARNING: PLAYING ONLINE BATTLE ARENA WITH DISCRETE-CONTINUOUS HYBRID ACTION SPACE
Most existing deep reinforcement learning (DRL) frameworks consider action spaces that are either discrete or continuous space. Motivated by the project of design Game AI for King of Glory (KOG), one the world’s most popular mobile game, we consider the scenario with the discrete-continuous hybrid action space. To directly apply existing DLR frameworks, existing approaches either approximate the hybrid space by a discrete set or relaxing it into a continuous set, which is usually less efficient and robust. In this paper, we propose a parametrized deep Q-network (P-DQN) for the hybrid action space without approximation or relaxation. Our algorithm combines DQN and DDPG and can be viewed as an extension of the DQN to hybrid actions. The empirical study on the game KOG validates the efficiency and effectiveness of our method.
 This paper examines a modified NN architecture and algorithm (P-DQN) for learning in hybrid discrete/continuous action spaces. The authors come up with a clever way of modifying the architecture of parameterized-action-space DDPG (as in Hausknecht & Stone 16) in such a way that the actor only outputs values for the continuous actions and the critic outputs values for all discrete actions, parameterized by the actor’s choice of continuous actions. Overall, I think this is an interesting and valid modification to the DDPG architecture, with results to show improved sample complexity. However, there is no quantitative analysis of why the new architecture works better, insufficient understanding of the new domain and learning task, and overall rough presentation. Clarity: The writing clarity is rough, but understandable, with numerous minor grammar mistakes. The paper is overly long and could be improved by a more compact presentation of background, algorithms, and results. Originality: The paper builds on DDPG and explores a novel modification to the architecture. Significance: It’s hard to evaluate the significance of this result because of the lack of videos + information on the Moba environment. The proposed P-DQN architecture is interesting and, if the results on the Moba environment are general, could be of use in future hybrid-discrete-continuous action space domains. Pros: • The modification to DDPG is genuinely interesting and does result in an algorithm that is a hybrid between DQN and DDPG. • The learning curves show evidence of faster learning using the P-DQN architecture. Cons: • It’s difficult to confidently evaluate the merits of P-DQN vs DDPG based only on learning curves from a single, new domain. It would be nice to have explored results on additional domains such as Robot Soccer (HFO), where algorithm could have been compared directly to DDPG. • There is very little analysis of why P-DQN exhibits better sample complexity. The authors claim the difference stems from explicit computation over the discrete actions, but this is never analyzed. • Very difficult to read the axes on the plots in Fig 3. • Not much detail is given about the domain – who or what is the agent playing against? Is the agent playing against a bot or just learning to kill creeps? Would be great to have a video of the learned policy (or evaluation against human / scripted opponent) so that others can understand the quality of the learned policy.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
PARAMETRIZED DEEP Q-NETWORKS LEARNING: PLAYING ONLINE BATTLE ARENA WITH DISCRETE-CONTINUOUS HYBRID ACTION SPACE
Most existing deep reinforcement learning (DRL) frameworks consider action spaces that are either discrete or continuous space. Motivated by the project of design Game AI for King of Glory (KOG), one the world’s most popular mobile game, we consider the scenario with the discrete-continuous hybrid action space. To directly apply existing DLR frameworks, existing approaches either approximate the hybrid space by a discrete set or relaxing it into a continuous set, which is usually less efficient and robust. In this paper, we propose a parametrized deep Q-network (P-DQN) for the hybrid action space without approximation or relaxation. Our algorithm combines DQN and DDPG and can be viewed as an extension of the DQN to hybrid actions. The empirical study on the game KOG validates the efficiency and effectiveness of our method.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 This paper examines a modified NN architecture and algorithm (P-DQN) for learning in hybrid discrete/continuous action spaces. The authors come up with a clever way of modifying the architecture of parameterized-action-space DDPG (as in Hausknecht & Stone 16) in such a way that the actor only outputs values for the continuous actions and the critic outputs values for all discrete actions, parameterized by the actor’s choice of continuous actions. Overall, I think this is an interesting and valid modification to the DDPG architecture, with results to show improved sample complexity. However, there is no quantitative analysis of why the new architecture works better, insufficient understanding of the new domain and learning task, and overall rough presentation. Clarity: The writing clarity is rough, but understandable, with numerous minor grammar mistakes. The paper is overly long and could be improved by a more compact presentation of background, algorithms, and results. Originality: The paper builds on DDPG and explores a novel modification to the architecture. Significance: It’s hard to evaluate the significance of this result because of the lack of videos + information on the Moba environment. The proposed P-DQN architecture is interesting and, if the results on the Moba environment are general, could be of use in future hybrid-discrete-continuous action space domains. Pros: • The modification to DDPG is genuinely interesting and does result in an algorithm that is a hybrid between DQN and DDPG. • The learning curves show evidence of faster learning using the P-DQN architecture. Cons: • It’s difficult to confidently evaluate the merits of P-DQN vs DDPG based only on learning curves from a single, new domain. It would be nice to have explored results on additional domains such as Robot Soccer (HFO), where algorithm could have been compared directly to DDPG. • There is very little analysis of why P-DQN exhibits better sample complexity. The authors claim the difference stems from explicit computation over the discrete actions, but this is never analyzed. • Very difficult to read the axes on the plots in Fig 3. • Not much detail is given about the domain – who or what is the agent playing against? Is the agent playing against a bot or just learning to kill creeps? Would be great to have a video of the learned policy (or evaluation against human / scripted opponent) so that others can understand the quality of the learned policy.
assistant
 This paper examines a modified NN architecture and algorithm (P-DQN) for learning in hybrid discrete/continuous action spaces. The authors come up with a clever way of modifying the architecture of parameterized-action-space DDPG (as in Hausknecht & Stone 16) in such a way that the actor only outputs values for the continuous actions and the critic outputs values for all discrete actions, parameterized by the actor’s choice of continuous actions. Overall, I think this is an interesting and valid modification to the DDPG architecture, with results to show improved sample complexity. However, there is no quantitative analysis of why the new architecture works better, insufficient understanding of the new domain and learning task, and overall rough presentation. Clarity: The writing clarity is rough, but understandable, with numerous minor grammar mistakes. The paper is overly long and could be improved by a more compact presentation of background, algorithms, and results. Originality: The paper builds on DDPG and explores a novel modification to the architecture. Significance: It’s hard to evaluate the significance of this result because of the lack of videos + information on the Moba environment. The proposed P-DQN architecture is interesting and, if the results on the Moba environment are general, could be of use in future hybrid-discrete-continuous action space domains. Pros: • The modification to DDPG is genuinely interesting and does result in an algorithm that is a hybrid between DQN and DDPG. • The learning curves show evidence of faster learning using the P-DQN architecture. Cons: • It’s difficult to confidently evaluate the merits of P-DQN vs DDPG based only on learning curves from a single, new domain. It would be nice to have explored results on additional domains such as Robot Soccer (HFO), where algorithm could have been compared directly to DDPG. • There is very little analysis of why P-DQN exhibits better sample complexity. The authors claim the difference stems from explicit computation over the discrete actions, but this is never analyzed. • Very difficult to read the axes on the plots in Fig 3. • Not much detail is given about the domain – who or what is the agent playing against? Is the agent playing against a bot or just learning to kill creeps? Would be great to have a video of the learned policy (or evaluation against human / scripted opponent) so that others can understand the quality of the learned policy.