LEARNING PROGRAMMATICALLY STRUCTURED REPRESENTATIONS WITH PERCEPTOR GRADIENTS
We present the perceptor gradients algorithm -a novel approach to learning symbolic representations based on the idea of decomposing an agent's policy into i) a perceptor network extracting symbols from raw observation data and ii) a task encoding program which maps the input symbols to output actions. We show that the proposed algorithm is able to learn representations that can be directly fed into a Linear-Quadratic Regulator (LQR) or a general purpose A* planner. Our experimental results confirm that the perceptor gradients algorithm is able to efficiently learn transferable symbolic representations as well as generate new observations according to a semantically meaningful specification.
The high-level problem this paper tackles is that of learning symbolic representations from raw noisy data, based on the hypothesis that symbolic representations that are grounded in the semantic content of the environment are less susceptible to overfitting.
The authors propose the perceptor gradients algorithm, which decouples the policy into 1) a perceptor network that maps raw observations to domain-specific representations, which are inputs to 2) a pre-specified domain-specific control or planning program. The authors claim that such a decomposition is general enough to accommodate any task encoding program.
The proposed method is evaluated on three experiments: a simple control task (cartpole-balancing), a navigation task (minecraft: go to pose), and a stochastic single-object retrieval task (minecraft: collect wood). The authors show that the perceptor gradients algorithm learns much faster than vanilla policy gradient. They also show that the program provides an inductive bias that helps ground the representations to the true state of the agent by manually inspecting the representations and by reconstructing the representation into a semantically coherent scene.
This paper is clear and well-written and I enjoyed reading it. It proposes a nice perspective of leveraging programmatic domain knowledge and integrating such knowledge with a learned policy for planning and control. If the following concerns were addressed I would consider increasing my score.
1. To what extent do the experiments support the authors' claims: Although the existing experiments are very illustrative and clear, they did not seem to me to illustrate that the learned representations are transferable as the authors claimed in the introduction. This perhaps is due to the ambiguous definition of "transferable;" it would be helpful if the authors clarified what they mean by this. Nevertheless, as the paper suggests in the introduction that symbolic representations are less likely to overfit to the training distribution, I would be interested to see an experiment that illustrates the capability of the program-augmented policy to generalize to new tasks. For example, Ellis et al. [1] suggested that the programs can be leveraged to extrapolate to problems not previously seen in the input (e.g. by running the for loop for more iterations). To show the transferability of such symbolic representations, is it possible for the authors to include an experiment to show to what extent the perceptor gradients algorithm can generalize to new problems? For example, is it possible for the proposed approach to train on "Minecraft: Go to Pose" and generalize to a larger map? Or is it possible for the proposed approach to train on one wood block and generalize to more wood blocks?
2. Experiment request: The paper seems to suggest that the "Minecraft: Go to Pose" task and the "Minecraft: Collect Wood" task were trained with an autoencoding perceptor. To more completely assess to what extent the program is responsible for biasing the representations to be semantically grounded in environment, would the authors please provide ablation experiments (learning curves, and visualization of the representations) for these two tasks where only the encoder was used?
3. Question: I am a bit confused by the beta-VAE results in Figure 4. If the beta-VAE is trained to not only reconstruct its input as well as perform a "linear regression between the learnt latent space and the ground truth values" (page 6), then I would have expected that the latent space representations to match the ground truth values much more closely. Would the authors be able to elaborate more on the training details and objective function of the beta-VAE and provide an explanation for why the learned latent space deviates so far from the ground truth?
5. Related work: 
    a) The paper briefly discusses representation learning in computer vision and physical dynamics modeling. However, in these same domains it lacks a discussion of approaches that do use programs to constrain learned representations, as in [1-3]. Without this discussion, my view is that the related work would be very incomplete because program-induced constraints are core to this paper. Can the authors please provide a more thorough and complete treatment of this area?
    b) The approaches that this paper discusses for representation learning have been around for quite a long time, but it seems rather a misrepresentation of the related work to have all but two citations in the Related Work section 2017 and after. For example, statistical constraints on the latent space have been explored in [4-5]. Can the authors please provide a more thorough and complete treatment of the related work?
6. Possible limitation: A potential limitation for decoupling the policy in this particular way is that if the perceptor network produced incorrect representations that are fed into the program, the program cannot compensate for these errors. It would be helpful for the authors to include a discussion about this in paper.
7. How does this scale? As stated in the intro, the motivation for this work is for enabling autonomous agents to learn from raw visual data. Though the experiments in this paper were illustrative of the approach, these experiments assumed that the agent had access to the true state variables of its environment (like position and velocity), and the perceptor network is just inferring the particular values of these variables for a particular problem instance. However, presumably the motivation for learning from raw visual data is that the agent does not have access to the simulator of the environment. How do the authors envision their proposed approach scaling to real world settings where the true state variables are unknown? There is currently not an experiment that shows a need for learning from raw visual data. This is a major concern, because if the only domains that the perceptor gradients algorithm can be applied are those where the agent already has access to the true state variables, then there may be no need to learn from pixels in the first place. This paper would be made significantly stronger with an experiment where 1) learning from raw visual data is necessary (for example, if it is the real world, or if the true state variables were unknown) and 2) where the inductive bias provided by the program helps significantly on that task in terms of learning and transfer. Such an experiment would decisively reflect the paper's claims.
8. Clarity: The paper mentions that it is possible to generate new observations from the latent space. The paper can be made stronger by a more motivated discussion of why generating new observations is desirable, beyond just as a visualization tool. For example, the authors may consider making a connection with the analysis-by-synthesis paradigm that characterizes the Helmholtz machine.
[1] Ellis et al. (https://arxiv.org/pdf/1707.09627.pdf)
[2] Wu et al. (http://papers.nips.cc/paper/6620-learning-to-see-physics-via-visual-de-animation.pdf)
[3] Kulkarni et al. (https://www.cv-foundation.org/openaccess/content_cvpr_2015/html/Kulkarni_Picture_A_Probabilistic_2015_CVPR_paper.html)
[4] Schmidhuber (ftp://ftp.idsia.ch/pub/juergen/factorial.pdf)
[5] Bengio et al. (https://arxiv.org/abs/1206.5538)