Recent work has demonstrated how predictive modeling can endow agents with rich knowledge of their surroundings, improving their ability to act in complex environments. We propose question-answering as a general paradigm to decode and understand the representations that such agents develop, applying our method to two recent approaches to predictive modeling -action-conditional CPC (Guo et al., 2018) and SimCore (Gregor et al., 2019). After training agents with these predictive objectives in a visually-rich, 3D environment with an assortment of objects, colors, shapes, and spatial configurations, we probe their internal state representations with a host of synthetic (English) questions, without backpropagating gradients from the question-answering decoder into the agent. The performance of different agents when probed in this way reveals that they learn to encode detailed, and seemingly compositional, information about objects, properties and spatial relations from their physical environment. Our approach is intuitive, i.e. humans can easily interpret the responses of the model as opposed to inspecting continuous vectors, and model-agnostic, i.e. applicable to any modeling approach. By revealing the implicit knowledge of objects, quantities, properties and relations acquired by agents as they learn, question-conditional agent probing can stimulate the design and development of stronger predictive learning objectives.
########## Post-Rebuttal Summary ###########
The authors engaged actively in the rebuttal discussion and in the process we were able to concretize the motivation of the submission (as a result in increased my score). However, I think that the submission in its current form lacks experimental analysis of the proposed conditional probes, especially the trade-offs on the reliability of the representation analysis when performed with a conditional probe as well as a clear motivation for the need of a language interface. I can therefore not recommend the submission in its current form for acceptance.
(for detailed discussions + suggested experiments please see rebuttal discussions)
#######################################
## Paper Summary
The paper tackles the problem of analyzing the information captured in the learned representation of a deep neural network. It proposes to replace the commonly used "probing networks" that try to directly infer the information from the learned representation, with a language interface in the form of a QA model which is trained post-hoc without propagating gradients into the learned representation. The authors argue that such a language interface provides a more natural interface for probing the information in a learned representation. The paper shows representation analysis results for the internal representations of agents trained on an exploratory task in a simple, simulated household environment similar to DM Lab. The authors conduct additional analysis on representations learned with auxiliary generative and contrastive objectives.
## Strengths
- the language of the paper is clear and easy to follow
- the paper covers the related work well
- the provided explanations help understand the content of the paper
- the analysis of how the information captured in the agent's representation changes over the course of a trajectory is interesting (more such visualizations in the appendix would be nice!)
## Weaknesses
(1) the motivation for the proposed problem does not convince: why do I need to train a Q/A system to infer which components of the true state are captured by the learned representation? For each of the properties of the environment I could train a probing network (as is done e.g. in [1,2]) and would get much more precise answers; ground truth labels need to be available whether I train QA or probing networks. The authors provide two motivations for the proposed approach which both do not convince me:
	(a) QA "provides an intuitive investigation tool for humans": I cannot imagine a workflow in which researchers would prefer to ask questions to their model over a plot showing explicit regression accuracies aggregated across many data samples (which the conventional probing networks provide).
	(b) "the space of questions is open-ended, [...] enabling comprehensive analysis of [an agent's] internal state": for each new question type we need to provide a sufficiently large number of question-answer-pairs to train the QA system for this question type. we could therefore also train a probing network using the same labels and would get a better overview of whether the state information is captured in the representation. I fail to see how substituting probing networks with a language interface helps the investigation of the representation's properties.
	
(2) the paper only provides minor novelty: the main proposal is to replace the probing networks, which were extensively used in prior work, with a natural language interface; a proposal that does not seem to provide a clear advantage (see above). The paper does not provide any further technical novelty.
	
(3) it is possible that the analysis of the contrastive model could improve substantially with a different sampling scheme of the negative examples: maybe sampling from different sequences makes the task of discriminating too easy so that the model is not encouraged to learn a rich representation. It would be interesting to see whether the representation captures more information if a more standard contrastive objective is chosen that discriminates between future frames with different offset in the same sequence (see for example the objective in [3]).
(4) it seems that both environment and chosen task will have high influence on the representation the agent can learn from the collected data. Therefore the fact that the authors evaluate their approach only on a single environment / task combination, both of which they introduce themselves, weakens any conclusions the authors draw from their experiments. It would help strengthen the message of the paper to apply their methodology to previously proposed environment / task combinations, for example in the AI2-THOR environment [4]
[Novelty]: minor
[technical novelty]: minor
[Experimental Design]: not comprehensive
[potential impact]: low
#########################
[overall recommendation]: Reject - In its current form the paper does not provide a convincing argument for why learning a language interface for probing a representation is better than learning the usual probing networks. Further there are some doubts on the setup of the contrastive objective and the paper lacks comprehensive evaluation on standard environments. Therefore I cannot recommend acceptance in its current form.
[Confidence]: High
[1] Neural Predictive Belief Representations, Guo et al., 2018
[2] Shaping Belief States with Generative Environment Models for Reinforcement Learning, Gregor et al., 2019
[3] Representation Learning with Contrastive Predictive Coding, Van den Oord et al., 2018
[4] AI2-THOR: An Interactive 3D Environment for Visual AI, Kolve et al., 2017
#### Final Rebuttal comment (to make it visible to the authors) ####
I understand the authors point that every method for probing representations can potentially be flawed in that the probing mechanism might not be expressive enough to extract the information that is indeed present in the representation. The point I was trying to make in my previous response is, that in the case of conditional probes that try to solve multiple such "probing tasks" in parallel, such uncertainties accumulate because the probing mechanism might trade off performance on one task for performance on another (if solving all of them at once is too challenging). If the submission's key contribution is to make probes conditional this seems like a trade-off that should be experimentally investigated, as it is vital to practitioners how much they can trust their probing method to extract the relevant information if it is actually in the representation.
One possible experiment would be to train an unconditional probe per attribute (maybe on a subset of all possible attribute-object combinations) and then a conditional probe across all of them to show that the conditional probe's assessments of the representation agree with the ensemble of unconditional probes. In my previous response I additionally raised the point that even if it can be shown that conditional probes are reliable, the authors still need to provide arguments why that conditioning should work via a language interface, not for example a symbolic/one-hot interface. If the claim is that this allows for generalization, it should again be shown that this generalization does not increase the error of the probe substantially, such that meaningful conclusions about the representation are still possible.
Regarding testing on more diverse environments: the questions these experiments are supposed to answer (i.e. how reliable are conditional probes) are inherently empirical, so verifying across diverse environments will make the analysis more conclusive.
I agree with the author's point that training 7k unconditional probes would not be practical, probably the current approach would be to train an image decoder that then reconstructs a top-down view of the whole scene. This, however, would have the same problems as a conditional probe, i.e. the probing decoder could trade-off performance between reconstructing different parts of the image. Therefore, I agree with the authors that investigating conditional probes is an interesting direction, but the submission does not provide a comprehensive analysis of this question.
On a final note, I appreciate the continued, factual discussion with the authors and think that the refinement of the focus towards conditional vs unconditional probes is a step in the right direction. To acknowledge that I am raising my score from "reject" to "weak reject". Yet, I think that the work in its current form lacks experimental analysis of the proposed conditional probes. During the rebuttal discussion I highlighted concerns and proposed possible experiments. I cannot recommend acceptance of the submission in its current form, but I encourage the authors to incorporate the refined motivation and add more comprehensive experimental evaluation for a possible resubmission.