Value-based reinforcement learning (RL) methods like Q-learning have shown success in a variety of domains. One challenge in applying Q-learning to continuous-action RL problems, however, is the continuous action maximization (max-Q) required for optimal Bellman backup. In this work, we develop CAQL, a (class of) algorithm(s) for continuous-action Q-learning that can use several plugand-play optimizers for the max-Q problem. Leveraging recent optimization results for deep neural networks, we show that max-Q can be solved optimally using mixed-integer programming (MIP). When the Q-function representation has sufficient power, MIP-based optimization gives rise to better policies and is more robust than approximate methods (e.g., gradient ascent, cross-entropy search). We further develop several techniques to accelerate inference in CAQL, which despite their approximate nature, perform well. We compare CAQL with state-of-the-art RL algorithms on benchmark continuous-control problems that have different degrees of action constraints and show that CAQL outperforms policy-based methods in heavily constrained environments, often dramatically.
Summary: 
This paper targets the maximization issue in continuous value based methods, especially Q Learning. The idea is to use Mixed Integer Programming (MIP) to solve the Q maximization step by formulating the neural network structure of the Q function as a constrained mixed integer program. Further improvements are made by approximating the MIP solution in order to make training/inference faster. The method is tested on tasks from the Mujoco domain and compared with other value based methods for continuous control. I found the paper simple to follow and well structured. The problem is well motivated too and the empirical analysis is quite rigorous.    
The obvious concerns are regarding scalability of the method; both in terms of 1) using other forms of neural network components (ex. Other activation functions, Convolutional networks in the case of vision based problems such as robotic manipulation) and 2) problems that are inherently less sample efficient, i.e. cases where shortening the sampling horizon is not a feasible option for learning meaningful policies. 
Overall, I feel the positive aspects more or less outweigh the drawbacks and therefore my vote is for a weak accept.
Comments/Questions: 
- Table 8 description says hyper-parameter sweeps were done for temperature and exploration noise decay values but the table is missing their values.
- What happens when action range is increased from default? One of the reasons mentioned for constraining the action space is to validate how well policy-based methods work. To really take this point home, I feel it might be good to check with an increased action range.
- Controlling w.r.t symmetry of the env (esp. in Mujoco domains), thus reducing the number of actions by half, might help faster MIP computation times. 
- Figure 3 x-axis runs till different values, but the description says training steps are 1000. How long are the experiments run for?
- Can the authors elaborate on why the episode length is decreased from 1000 to 200?