In this paper, we formulate hypothesis verification as a reinforcement learning problem. Specifically, we aim to build an agent that, given a hypothesis about the dynamics of the world can take actions to generate observations which can help predict whether the hypothesis is true or false. Our first observation is that agents trained end-to-end with the reward fail to learn to solve this problem. In order to train the agents, we exploit the underlying structure in the majority of hypothesesthey can be formulated as triplets (pre-condition, action sequence, post-condition). Once the agents have been pretrained to verify hypotheses with this structure, they can be fine-tuned to verify more general hypotheses. Our work takes a step towards a "scientist agent" that develops an understanding of the world by generating and testing hypotheses about its environment.
This paper trains agents which are able to verify hypotheses, such as “the blue switch causes the door to open”. It does this by first pretraining the agent to perform interventions in the environment which change the states of the objects of interest, and then finetuning the agent to actually make a decision about whether the given hypothesis is correct. The paper shows that agents trained using this procedure are able to not only verify the types of hypotheses seen during pre-training, but also learn to verify more complex hypotheses. In contrast, an agent which is trained directly on the hypothesis verification task is unable to learn to do it. 
Overall, I enjoyed reading this paper and thought that it provided an interesting take on the question of how to train agents that can appropriately gather information about their environments. However, (1) the paper lacks any discussion of related work in terms of causal reasoning and partial observability, and (2) the experiments and analysis seem weak. I thus am giving a score of “weak reject”, though it is possible I could increase my score is some of my concerns can be addressed.
First, I was very surprised to see that the paper included no discussion at all about either causal reasoning or partial observability. The whole notion of verifying hypotheses—particularly those in the triplet form as presented in the paper—is equivalent to the idea of performing inference about the structure of a causal graph with three variables. The choice of which interventions to perform in order to make these inferences is a well-studied problem [1] and has been recently explored in the context of RL as well [2]. The novelty here seems to be in embedding the problem of causal reasoning in harder credit assignment problems (i.e. longer time horizon), though see [3]. Similarly, the setup of the MDP in the paper is actually a POMDP, where the state includes the truth value of the hypothesis but where observations do not include this information. Yet, there is no mention of POMDPs or discussion of the literature on partial observability in the paper.
 
Second, I felt that the setup was overly complex in places making it difficult to draw conclusions, that there were a lack of comparisons, and that the analysis was not as in depth as it could have been. For example, why is it necessary to represent the hypothesis with natural language? Why not use a symbolic representation? It seems like including the pseudo-natural language adds unnecessary complexity and makes it difficult to distentangle what about the problem is hard (Understanding the hypothesis? Choosing the right interventions? Parsing the observations correctly?). The utility of having it be closer to language is that you might see generalization between related hypotheses, but this isn’t really something that is actually tested for since all hypotheses are trained on either during pretraining or finetuning.
 
I also feel like the choice of pretraining reward feels somewhat arbitrary, and it would have been nice to see comparisons to other alternatives (and even better, to other forms of intrinsic motivation). For example, here are a few alternate ways of rewarding the agent that seem intuitively like they could also work:
Reward the agent for changing the state of any of the objects in the environment
Reward the agent for changing the state of any object referenced in the hypothesis
Reward the agent for observing a state of the world it has not seen before (i.e. count-based exploration)
In other words, how important is the fact that the reward is given based on the pre and postconditions?
I thought the paper would benefit from more detailed analyses to tease apart the behavior of the agent. For example, I am curious how many errors are a result of errors in the predictor versus poor exploration behavior by the policy. Could you report (1) how frequently the policy’s behavior results in the right observations necessary to make a decision, and (2) results with a policy which uses an oracle predictor (i.e. which will always report the correct answer, if there was enough data in the last N frames to detect that answer)?
On the more practical side, I also thought the quality of the evaluations was not very thorough. For instance, it looks like the pretraining proceeds for 1e8 steps and finetuning for 5e7 steps, based on the plots (these values should be stated more explicitly in the paper). However, this is a bit of an unfair comparison for the “RL Baseline”, as it only is trained for 5e7 steps while the other agents are trained for 1.5e8 steps. I would like to see a comparison where the RL Baseline agent is trained for 1.5e8 steps as well. Similarly, on the bottom of page 6 the paper says “we show the max out of five for each of the methods shown”. However, only reporting the max value is considered bad practice and can result in misleading comparisons (see Joelle Pineau’s talk on “Reproducible, Reusable, and Robust Reinforcement Learning” at NeurIPS 2018). I’d like to see the data in all figures and tables reported with means or medians across seeds, rather than best seeds.
A few minor comments:
- Please state in the main text which RL algorithm you use.
- Can you clarify whether Figure 2 show the proxy rewards or the true rewards?
- For R_pre and R_ppost, what values do you use for C and N?
[1] Pearl, J. (2000). Causality: models, reasoning and inference (Vol. 29). Cambridge: MIT press.
[2] Dasgupta, I., Wang, J., Chiappa, S., Mitrovic, J., Ortega, P., Raposo, D., ... & Kurth-Nelson, Z. (2019). Causal reasoning from meta-reinforcement learning. arXiv preprint arXiv:1901.08162.
[3] Denil, M., Agrawal, P., Kulkarni, T. D., Erez, T., Battaglia, P., & de Freitas, N. (2016). Learning to perform physics experiments via deep reinforcement learning. arXiv preprint arXiv:1611.01843.
--
Update after rebuttal:
Thank you very much for your response. However, I do not feel that all of my concerns have been addressed and thus will keep my score as it is. In particular, I still feel the paper lacks sufficient discussion of the literature on causal reasoning. I also do not think it is sufficient to add an appendix with the results across multiple seeds: these results should be in the main paper. I'm not sure I follow the justification that max seeds make sense because "the reward distribution is quite binary in nature"---the plots shown in Figure 3 and 4, for example, span a range of values from 0 to 1. I find the plots that have both variance and max seed very hard to interpret---in some cases the mean is so much lower than the max seed that the variance region doesn't overlap at all. More broadly, it might be easier to compare using bar plots showing final performance, rather than training curves
I appreciate the additional results, especially with different pretraining schemes---thanks for adding these! I have a bit of hard time interpreting the results though since there are no direct comparisons with the triplet pretraining scheme; it would be helpful if these results could be included in these figures too.