We study the problem of deep reinforcement learning where the agent's action sequences are constrained, e.g., prohibition of dithering or overactuating action sequences that might damage a robot, drone, or other physical device. Our model focuses on constraints that can be described by automata such as DFAs or PDAs. We then propose multiple approaches to augment the state descriptions of the Markov decision process (MDP) with summaries of recent action histories. We empirically evaluate these methods applying DQN to three Atari games, training with reward shaping. We found that our approaches are effective in significantly reducing, and even eliminating, constraint violations while maintaining high reward. We also observed that the total reward achieved by an agent can be highly sensitive to how much the constraints encourage or discourage exploration of potentially effective actions during training, and, in addition to helping ensure safe policies, the use of constraints can enhance exploration during training.
This work aims to use formal languages to add a reward shaping signal in the form of a penalty on the system when constraints are violated. There is also an interesting notion of using an embedding based on the action history to aid the agent in avoiding violations. However, I do not believe this paper did a good enough job in situating this work in the context of prior work — in particular (Camacho 2017). There is a significant related work section that does an ok job of describing many other works, but to my knowledge (Camacho 2017) is the most similar to this one (minus the embedding), yet is not mentioned here. It is difficult to find all related work of course, so I would encourage revision with detailed description of the novelty of this work in comparison with that one. I would also encourage an more thoughtful examination of the theoretical ramifications of the reward shaping signal with respect to the optimal policy as (Camacho 2017) do and as is modeled in the (Ng 1999) paper. As of this revision, however, I'm not sure I would recommend it for publication. Additionally, I suggest that the authors describe the reward shaping mechanism a bit more formally, it was unclear whether it fits into Ng's potential function methodology at first pass.
Comments:
+ It would be nice to explain to the reader in intuitive terms what “no-1D-dithering” means near this text. I understand that later on this is explained, but for clarity it would be good to have a short explanation during the first mentioning of this term as well.
+ It would be good to clarify in Figure 1 what . * (lr)^2 is since in the main text near the figure is is just (lr)^2 and the .* is only explained several pages ahead
+ An interesting connection that might be made is that Ng et al.’s reward shaping mechanism, if the shaping function is based on a state-dependent potential then the optimal policy under the new MDP is still optimal for the old MDP. It would be interesting to see how well this holds under this holds under this schema. In fact, this seems like analysis that several other works have done for a very similar problem (see below).
+ I have concerns about the novelty of this method. It seems rather similar to 
Camacho, Alberto, Oscar Chen, Scott Sanner, and Sheila A. McIlraith. "Decision-making with non-markovian rewards: From LTL to automata-based reward shaping." In Proceedings of the Multi-disciplinary Conference on Reinforcement Learning and Decision Making (RLDM), pp. 279-283. 2017.
Camacho, Alberto, Oscar Chen, Scott Sanner, and Sheila A. McIlraith. "Non-Markovian Rewards Expressed in LTL: Guiding Search Via Reward Shaping." In Proceedings of the Tenth International Symposium on Combinatorial Search (SoCS), pp. 159-160. 2017.
However, that work proposes a similar framework in a much more formal way. In fact, in that work also a DFA is used as a reward shaping signal -- from what I can tell for the same purpose through a similar mechanism. It is possible, however, that I missed something which contrasts the two works.
Another work that can be referenced:
De Giacomo, Giuseppe, Luca Iocchi, Marco Favorito, and Fabio Patrizi. "Reinforcement Learning for LTLf/LDLf Goals." arXiv preprint arXiv:1807.06333 (2018).
I think it is particularly important to situate this work within the context of those others. 
+ General the structure of the paper was a bit all over the place, crucial details were spread throughout and it took me a couple of passes to put things together. For example, it wasn't quite clear what the reward shaping mechanism was until I saw the -1000 and then had to go back to figure out that basically -1000 is added to the reward if the constraint is violated. I would suggest putting relevant details all in one place. For example, "Our reward shaping function F(x) was  { -1000, constraint violation, 0 otherwise}".