It has long been assumed that high dimensional continuous control problems cannot be solved effectively by discretizing individual dimensions of the action space due to the exponentially large number of bins over which policies would have to be learned. In this paper, we draw inspiration from the recent success of sequence-to-sequence models for structured prediction problems to develop policies over discretized spaces. Central to this method is the realization that complex functions over high dimensional spaces can be modeled by neural networks that predict one dimension at a time. Specifically, we show how Q-values and policies over continuous spaces can be modeled using a next step prediction model over discretized dimensions. With this parameterization, it is possible to both leverage the compositional structure of action spaces during learning, as well as compute maxima over action spaces (approximately). On a simple example task we demonstrate empirically that our method can perform global search, which effectively gets around the local optimization issues that plague DDPG. We apply the technique to off-policy (Q-learning) methods and show that our method can achieve the state-of-the-art for off-policy methods on several continuous control tasks.
The paper describes a new RL technique for high dimensional action spaces.  It discretizes each dimension of the action space, but to avoid an exponential blowup, it selects the action for each dimension in sequence.  This is an interesting approach.  The paper reformulates the MDP with a high dimensional action space into an equivalent MDP with more time steps (one per dimension) that each selects the action in one dimension.  This makes sense.
While I do like very much the model, I am perplex about the training technique.  The lower MDP is precisely the new proposed model with unidimensional actions and therefore it should be sufficient.  However, the paper also describes an upper MDP that seems to be superfluous.  The two MDPs are mathematically equivalent, but their Q-values are obtained differently (TD-0 for the upper MDP and Q-learning for the lower MDP) and yet the paper tries to minimize the Euclidean distance between them.  This is really puzzling since the different training algorithms suggest that the Q-values should be different while minimizing the Euclidean distance between them tries to make them equal.  The paper suggests that divergence occurs without the upper MDP.  This is really suspicious. The approach feels like a band-aid solution to cover a problem that the authors could not identify.  While the empirical results are good, I don't think the paper should be published until the authors figure out a principled way of training.
The proposed approach reformulates the MDP with high dimensional actions into an equivalent one with uni dimensional actions.  There is a catch.  This approach effectively hides the exponential action space into the state space which becomes exponential.  Since u contains all the actions of the previous dimensions, we are effectively increasing the state space by an exponential factor.  The paper should discuss this and explain what are the consequences in practice.  In the end, the MDP does not become simpler.
Overall, this is an interesting paper with a good idea, but the training technique is not mature enough for publication.