As deep reinforcement learning driven by visual perception becomes more widely used there is a growing need to better understand and probe the learned agents. Understanding the decision making process and its relationship to visual inputs can be very valuable to identify problems in learned behavior. However, this topic has been relatively under-explored in the research community. In this work we present a method for synthesizing visual inputs of interest for a trained agent. Such inputs or states could be situations in which specific actions are necessary. Further, critical states in which a very high or a very low reward can be achieved are often interesting to understand the situational awareness of the system as they can correspond to risky states. To this end, we learn a generative model over the state space of the environment and use its latent space to optimize a target function for the state of interest. In our experiments we show that this method can generate insights for a variety of environments and reinforcement learning methods. We explore results in the standard Atari benchmark games as well as in an autonomous driving simulator. Based on the efficiency with which we have been able to identify behavioural weaknesses with this technique, we believe this general approach could serve as an important tool for AI safety applications.
Summary
This paper proposes a generative technique to sample "interesting" states useful for analyzing the behavior of deep reinforcement learning agents. In this context, the concept of "interesting" is defined via user-specific target functions, e.g. states that arise as a consequence of taking specific actions (such as actions associated with high or low Q-values for example). The approach is evaluated in the Atari domain and in an autonomous driving simulator. Results are mainly presented as visualizations of interesting states that are described verbally.
Quality
The quality of the submission is extremely low. The optimization objectives chosen by the authors seem very ad hoc to me and how the motivation relates to the objectives is hard to comprehend (see my Clarity section). The experimental results have very low quality as well---results are mainly depicted as images with a verbal explanation.
Clarity
The clarity of the paper is extremely poor. While I do conceptually understand Section 2.1, I have a hard time linking it precisely to Section 2.2. Just some examples regarding lack in clarity:
- What is z in Section 2.1?
- How do the objectives in Section 2.1 and Section 2.2 relate to each other, i.e. how does the algorithm operate? Some pseudocode would be really helpful here.
- What are the target functions S^+, S^- and S^\pm in Section 2.3?
- What is the difference in the KL-regularizer mentioned in the text below Equation (3) and in Equation (5)?
- In the text above Equation (2), it is mentioned that a squared reconstruction loss is insensitive to small elements in the image that have a huge impact on the reward. While this is true, I don't see how the multiplicative policy gradient norm term in Equation (2), as proposed by the authors, is addressing this issue. The proposed modification puts emphasis on states where the norm of the policy gradient is high, which is different from putting emphasis on specific regions in the image. I guess the intention would be to do an element-wise multiplication of the squared loss vector and the absolute value policy gradient vector before collapsing to a scalar, or something similar?
In general, I found the entire writing from Section 3 onward a bit wordy and I do not think that nine pages are required to deliver the message of the paper in its current form.
Originality
The idea of visualizing states that reveal interesting insights about an agent's behavior based on a user-defined target function sounds interesting. But I have not worked in interpretability of agent behavior, which is why I leave the assessment of the originality to the other reviewers and the area chair.
Significance
If the results of the paper were backed up with some proper scientific metrics other than verbally explaining images, there might be some significance in the paper.
Update
After the authors' response, I am currently not inclined to change my score. While I do agree that the paper proposes an interesting idea, the technical presentation of the work is simply too poor and not convincing at this stage. Here are a few points:
- A variational autoencoder works as follows. There is a generative model over latent variables z and observed variables s, consisting of a prior for z and a likelihood for s conditioned on z. The prior over z can be e.g. a normal distribution denoted as N(z|\mu_prior, \Sigma_prior). Then the likelihood (decoder) can be e.g. a deep neural net that maps z to a normal distribution of s denoted as N(s|\mu_likelihood(z, \theta), \Sigma_likelihood(z, \theta)) where \theta refers to the decoder's neural network weights. Furthermore, there is a recognition model that approximates the posterior over z given s (encoder)---this can also be e.g. a deep neural net that maps s to a normal distribution in z denoted as N(z|\mu_posteriorapprox(s, \psi), \Sigma_posteriorapprox(s, \psi)) where \psi refers to the encoder's neural network weights.
- The reparameterization trick is not required for the technical explanation of the involved random variables and how they relate to each other. It is merely an optimization trick to establish a functional dependency between a random variable and the parameters of its distribution (e.g. mean and covariance in the Gaussian case).
- To be specific about your updated paper. The notation you chose for the encoder f(s) = (\mu, \sigma) is confusing because it hides the dependency on s on the right hand side. The notation for the decoder g(\mu, \sigma, z) is also confusing because the decoder is supposed to map z to something in s space (see my first bullet point). The notation g(f(s), z) is particularly confusing because it is not consistent with the other notation that you use (which I mentioned in the sentence before). Usually, f(s) represents an element in latent space and is fed through the decoder to yield something in s space---so I don't understand why the decoder receives both f(s) and z as an argument.
- You talk about optimization objectives, then please specify what the optimization arguments are---this is not clear from the description given. For example, in Equation (1) the optimization arguments seem to be both the recognition (encoder) and the generative (prior over latents plus decoder) model parameters? In Equation (4), the optimization argument seems to be the latent variable z?
Given all the comments above, it is pretty obvious that the paper in its current form simply does not adhere to scientific standards for technically reporting machine learning algorithms in a proper way. So I clearly still vote for rejection because of the lack in technical clarity. And yes, as I said, I would like to see pseudocode.