Most existing deep reinforcement learning (DRL) frameworks consider action spaces that are either discrete or continuous space. Motivated by the project of design Game AI for King of Glory (KOG), one the world's most popular mobile game, we consider the scenario with the discrete-continuous hybrid action space. To directly apply existing DLR frameworks, existing approaches either approximate the hybrid space by a discrete set or relaxing it into a continuous set, which is usually less efficient and robust. In this paper, we propose a parametrized deep Q-network (P-DQN) farmework for the hybrid action space without approximation or relaxation. Our algorithm combines DQN and DDPG and can be viewed as an extension of the DQN to hybrid actions. The empirical study on the game KOG validates the efficiency and effectiveness of our method.
This paper presents a new reinforcement learning approach to handle environments with a mix of discrete and
continuous action spaces. The authors propose a parameterized deep Q-network (P-DQN) and leverage learning
schemes from existing algorithms such as DQN and DDPG to train the network. The proposed loss function and
alternating optimization of the parameters are pretty intuitive and easy to follow. My main concern is
with lack of sufficient depth in empirical evaluation and analysis of the method.
Pros:
1. The setup is an interesting and practically useful one to investigate. Many real-world environments require individual actions
 that are further parameterized over a continuous space.
2. The proposed method is simple and intuitive.
Cons:
1. The evaluation is performed only on a single environment in a restricted fashion. I understand the authors are restricted in the choice of environments which require a hybrid action space. However,
 even domains like Atari could be used in a setting where the continuous parameter x_k refers to the number of
 repetitions for action k. This is similar to the work of Lakshminarayanan et al. (2017). Could you test your algorithm in such a setting?
2. Comparison of the algorithm is performed only against DDPG. Have you tried other options like PPO (Schulman et al., 2017)?
 Also, considering that the action space is simplified in the experimental setup ("we use the default parameters of skills provided by the game environment, usually pointing to
the opponent hero's location"), with only the move(\alpha) action being a hybrid, one could imagine discretizing the move
direction \alpha and training a DQN (or any other algorithms over discrete action spaces) as another baseline.
3. The reward structure seems to be highly engineered. With so many components in the reward, it is not clear
what the individual contributions are and what policies are actually learned.
4. The authors don't provide any analysis of the empirical results. Do the P-DQN and DDPG converge to the same policy?
What factor(s) contribute most to the faster learning of P-DQN? Do the values of \alpha and \beta for the two-timescale
updates affect the results considerably?
5. (minor) The writing contains a lot of grammatical errors which makes this draft below par for an ICLR paper.
Other Questions:
1. In eq. 5.3, the loss over \theta is defined as the sum of Q values over different k. Did you try other formulations of
the loss? (say, product of the Q values for instance) One potential issue with the sum could be that if some values of k dominate this sum, Q(s, k, x_k; w) might not be maximized for all k.
2. Some terms of the reward function seem to be overly dependent on historic actions (ex. difference in gold and hitpoints). This could swamp the
influence of the other terms which are more dependent on the current action a_t, which might be an issue, especially with the Markovian assumption?
References:
 Lakshminarayanan et al, 2017; Dynamic Action Repetition for Deep Reinforcement Learning; AAAI
 Schulman et al., 2017; Proximal Policy Optimization Algorithms; Arxiv