title: Understanding and Diagnosing Deep Reinforcement Learning
summary: Deep neural policies have recently been installed in a diverse range of
  settings, from biotechnology to automated financial systems. However, the
  utilization of deep neural networks to approximate the value function leads to
  concerns on the decision boundary stability, in particular, with regard to the
  sensitivity of policy decision making to indiscernible, non-robust features due
  to highly non-convex and complex deep neural manifolds. These concerns
  constitute an obstruction to understanding the reasoning made by deep neural
  policies, and their foundational limitations. Hence, it is crucial to develop
  techniques that aim to understand the sensitivities in the learnt
  representations of neural network policies. To achieve this we introduce a
  theoretically founded method that provides a systematic analysis of the
  unstable directions in the deep neural policy decision boundary across both
  time and space. Through experiments in the Arcade Learning Environment (ALE),
  we demonstrate the effectiveness of our technique for identifying correlated
  directions of instability, and for measuring how sample shifts remold the set
  of sensitive directions in the neural policy landscape. Most importantly, we
  demonstrate that state-of-the-art robust training techniques yield learning of
  disjoint unstable directions, with dramatically larger oscillations over time,
  when compared to standard training. We believe our results reveal the
  fundamental properties of the decision process made by reinforcement learning
  policies, and can help in constructing reliable and robust deep neural
  policies.
paper_authors._id: 667d389dd2129316d13816a9
paper_authors.name: Ezgi Korkmaz
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-06-27T10:02:06.573Z
paper_authors._id: 667c1a5acb6800a191024eb9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/667c1a5acb6800a191024eb9/AqL8mQZsZjpZKi9FxtkIH.png
paper_authors.fullname: Ezgi Korkmaz
paper_authors.type: user
paper_authors.user: ezgikorkmaz
paper_title: Understanding and Diagnosing Deep Reinforcement Learning
paper_summary: Deep neural policies have recently been installed in a diverse range of
  settings, from biotechnology to automated financial systems. However, the
  utilization of deep neural networks to approximate the value function leads to
  concerns on the decision boundary stability, in particular, with regard to the
  sensitivity of policy decision making to indiscernible, non-robust features due
  to highly non-convex and complex deep neural manifolds. These concerns
  constitute an obstruction to understanding the reasoning made by deep neural
  policies, and their foundational limitations. Hence, it is crucial to develop
  techniques that aim to understand the sensitivities in the learnt
  representations of neural network policies. To achieve this we introduce a
  theoretically founded method that provides a systematic analysis of the
  unstable directions in the deep neural policy decision boundary across both
  time and space. Through experiments in the Arcade Learning Environment (ALE),
  we demonstrate the effectiveness of our technique for identifying correlated
  directions of instability, and for measuring how sample shifts remold the set
  of sensitive directions in the neural policy landscape. Most importantly, we
  demonstrate that state-of-the-art robust training techniques yield learning of
  disjoint unstable directions, with dramatically larger oscillations over time,
  when compared to standard training. We believe our results reveal the
  fundamental properties of the decision process made by reinforcement learning
  policies, and can help in constructing reliable and robust deep neural
  policies.
paper_ai_summary: A method is introduced to analyze the unstable decision boundaries of deep neural policies, revealing fundamental properties and helping construct robust reinforcement learning policies.
paper_ai_keywords: deep neural policies decision boundary stability indiscernible features non-convex manifolds sensitive directions deep neural manifolds Arcade Learning Environment (ALE) correlated directions robust training oscillations reinforcement learning policies