Many practical reinforcement learning problems contain catastrophic states that the optimal policy visits infrequently or never. Even on toy problems, deep reinforcement learners periodically revisit these states, once they are forgotten under a new policy. In this paper, we introduce intrinsic fear, a learned reward shaping that accelerates deep reinforcement learning and guards oscillating policies against periodic catastrophes. Our approach incorporates a second model trained via supervised learning to predict the probability of imminent catastrophe. This score acts as a penalty on the Q-learning objective. Our theoretical analysis demonstrates that the perturbed objective yields the same average return under strong assumptions and an -close average return under weaker assumptions. Our analysis also shows robustness to classification errors. Equipped with intrinsic fear, our DQNs solve the toy environments and improve on the Atari games Seaquest, Asteroids, and Freeway.
The paper addresses the problem of learners forgetting rare states and revisiting catastrophic danger states. The authors propose to train a predictive ‘fear model’ that penalizes states that lead to catastrophes. The proposed technique is validated both empirically and theoretically. 
Experiments show a clear advantage during learning when compared with a vanilla DQN. Nonetheless, there are some criticisms than can be made of both the method and the evaluations:
The fear radius threshold k_r seems to add yet another hyperparameter that needs tuning. Judging from the description of the experiments this parameter is important to the performance of the method and needs to be set experimentally. There seems to be no way of a priori determine a good distance as there is no way to know in advance when a catastrophe becomes unavoidable. No empirical results on the effect of the parameter are given.
The experimental results support the claim that this technique helps to avoid catastrophic states during initial learning.The paper however, also claims to address the longer term problem of revisiting these states once the learner forgets about them, since they are no longer part of the data generated by (close to) optimal policies.  This problem does not seem to be really solved by this method. Danger and safe state replay memories are kept, but are only used to train the catastrophe classifier. While the catastrophe classifier can be seen as an additional external memory, it seems that the learner will still drift away from the optimal policy and then need to be reminded by the classifier through penalties. As such the method wouldn’t prevent catastrophic forgetting, it would just prevent the worst consequences by penalizing the agent before it reaches a danger state. It would therefore  be interesting to see some long running experiments and analyse how often catastrophic states (or those close to them) are visited. 
Overall, the current evaluations focus on performance and give little insight into the behaviour of the method. The paper also does not compare to any other techniques that attempt to deal with catastrophic forgetting and/or the changing state distribution ([1,2]).
In general the explanations in the paper often often use confusing and  imprecise language, even in formal derivations, e.g.  ‘if the fear model reaches arbitrarily high accuracy’ or ‘if the probability is negligible’.
It is wasn’t clear to me that the properties described in Theorem 1 actually hold. The motivation in the appendix is very informal and no clear derivation is provided. The authors seem to indicate that a minimal return can be guaranteed because the optimal policy spends a maximum of epsilon amount of time in the catastrophic states and the alternative policy simply avoids these states. However, as the alternative policy is learnt on a different reward, it can have a very different state distribution, even for the non-catastrophics states.  It might attach all its weight to a very poor reward state in an effort to avoid the catastrophe penalty. It is therefore not clear to me that any claims can be made about its performance without additional assumptions.
It seems that one could construct a counterexample using a 3-state chain problem (no_reward,danger, goal) where the only way to get to the single goal state is to incur a small risk of visiting the danger state. Any optimal policy would therefore need to spend some time e in the danger state, on average. A policy that learns to avoid the danger state would then also be unable to reach the goal state and receive rewards. E.g pi* has stationary distribution (0,e,1-e) and return 0*0+e*Rmin + (1-e)*Rmax. By adding a sufficiently high penalty, policy pi~ can learn to avoid the catastrophic state with distribution (1,0,0) and then gets return 1*0+ 0*Rmin+0*Rmax= 0 < n*_M - e (Rmax - Rmin) = e*Rmin + (1-e)*Rmax - e (Rmax - Rmin).  This seems to contradict the theorem. It wasn’t clear what assumptions the authors make to exclude situations like this.
[1] T. de Bruin, J. Kober, K. Tuyls and R. Babuška, "Improved deep reinforcement learning for robotics through distribution-based experience retention," 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Daejeon, 2016, pp. 3947-3952.
[2] Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., ... & Hassabis, D. (2017). Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 201611835.