With the recent proliferation of the usage of reinforcement learning (RL) agents for solving real-world tasks, safety emerges as a necessary ingredient for their successful application. In this paper, we focus on ensuring the safety of the agent while making sure that the agent does not cause any unnecessary disruptions to its environment. The current approaches to this problem, such as manually constraining the agent or adding a safety penalty to the reward function, can introduce bad incentives. In complex domains, these approaches are simply intractable, as they require knowing apriori all the possible unsafe scenarios an agent could encounter. We propose a model-based approach to safety that allows the agent to look into the future and be aware of the future consequences of its actions. We learn the transition dynamics of the environment and generate a directed graph called the imaginative module. This graph encapsulates all possible trajectories that can be followed by the agent, allowing the agent to efficiently traverse through the imagined environment without ever taking any action in reality. A baseline state, which can either represent a safe or an unsafe state (based on whichever is easier to define) is taken as a human input, and the imaginative module is used to predict whether the current actions of the agent can cause it to end up in dangerous states in the future. Our imaginative module can be seen as a "plug-and-play" approach to ensuring safety, as it is compatible with any existing RL algorithm and any task with discrete action space. Our method induces the agent to act safely while learning to solve the task. We experimentally validate our proposal on two gridworld environments and a self-driving car simulator, demonstrating that our approach to safety visits unsafe states significantly less frequently than a baseline.
This paper presents a model-based approach to safety in RL, where the agent uses a transition model to plan ahead to avoid actions that can lead it to unsafe states. They call the planning component an imaginative module. The agent takes the baseline state as input - that can be used to define either a safe or unsafe state, that is used in the planning component. The authors claim that using these two techniques they can tackle both the safe exploration (not violating safety constraints during learning) and irreversible side-effects (unintended irreversible behavior due to poorly designed reward-function).  They validate their approach on two grid world environments and self-driving car simulators.
 
This paper should be rejected because of the assumptions it makes goes against the very task they are trying to solve. In the sense, the task is trivial given the assumptions they have. 
1) The inconsistent assumption regarding the access to trajectories to learn a model. 
The authors start with the assumption that the agent does not have access to the model (Sec 3) , and they explicitly learn the model. However, in the very next section (Sec 4), the authors assume that they can deploy a number of agents that interact with the environment randomly and collect that data to learn a complete transition model. Note that this assumption is wrong because:
If the random data agents are “safe”, i.e., don’t violate any safety constraint or cause any harmful behavior in the environment, then it is equivalent to assuming the agent having access to all the data to learn the model. This is a very big assumption that essentially says the agent has access to the model, which defeats the purpose of the safe-exploration problem. 
If the random agents are “unsafe”, i.e., they can violate the safety constraint, then it goes against the very claim made about their method being able to respect the constraints throughout the learning process. 
2) The assumption about the baseline state(s).  
This is also a pretty big assumption to have, that is not acknowledged in the paper. If the agent already has the set of all the states it needs to avoid (or the set of states that are safe), then along with the assumption regarding access to the model, solving reversibility is significantly easier task then the general safe exploration problem [1, 2]
3) The results reported in Figure 4 are not statistically significant. The experiments are only run over 3 random seeds [3] 
4) Can you give a few more details about the assumptions? In terms of how realistic they are or how essential they are to the method.
Things to improve the paper that did not impact the score:
- The negative side-effects problem that this work address is only based on reversibility criteria. 
Claim about learning the dynamics model is sample efficient is unsupported. 
References: 
[1] Berkenkamp, Felix, et al. "Safe model-based reinforcement learning with stability guarantees." Advances in neural information processing systems. 2017.
[2] Dalal, Gal, et al. "Safe exploration in continuous action spaces." arXiv preprint arXiv:1801.08757 (2018).
[3] Henderson, Peter, et al. "Deep reinforcement learning that matters." Thirty-Second AAAI Conference on Artificial Intelligence. 2018.