Unsupervised representation learning has succeeded with excellent results in many applications. It is an especially powerful tool to learn a good representation of environments with partial or noisy observations. In partially observable domains it is important for the representation to encode a belief state-a sufficient statistic of the observations seen so far. In this paper, we investigate whether it is possible to learn such a belief representation using modern neural architectures. Specifically, we focus on one-step frame prediction and two variants of contrastive predictive coding (CPC) as the objective functions to learn the representations. To evaluate these learned representations, we test how well they can predict various pieces of information about the underlying state of the environment, e.g., position of the agent in a 3D maze. We show that all three methods are able to learn belief representations of the environment-they encode not only the state information, but also its uncertainty, a crucial aspect of belief states. We also find that for CPC multi-step predictions and action-conditioning are critical for accurate belief representations in visually complex environments. The ability of neural representations to capture the belief information has the potential to spur new advances for learning and planning in partially observable domains, where leveraging uncertainty is essential for optimal decision making.
** Summary **
The authors evaluate three different representation learning algorithms for partially observable environments. In particular, they investigate how well the learned representation encodes the true belief distribution, including its uncertainty. 
They propose an extension to a previous algorithm and evaluate all three algorithms on a range of tasks.
** Clarity **
The paper is well written and overall easy to follow.
** Quality **
The paper evaluates the described algorithms on a sufficiently large set of tasks. There is no theoretical analysis.
** Originality & Significance **
While the authors propose a novel extension to an existing algorithm, I believe the value of this work lies in the detailed empirical analysis.
** Missing Citations **
I believe two recent papers (this year's ICML) should be mentioned in the related work section as they propose two representation learning algorithms for POMDPs that, as far as I can tell, are not yet mentioned in the paper but quite relevant to the discussed topic. [1] Because it also uses PSRs and [2] because it explicitly learns a belief state. It would be interesting to see how [2] compares in terms of performance to FP and CPC(|Action).
[1] Hefny, A., Marinho, Z., Sun, W., Srinivasa, S. & Gordon, G.. (2018). Recurrent Predictive State Policy Networks. Proceedings of the 35th International Conference on Machine Learning, in PMLR 80:1949-1958
[2] Igl, M., Zintgraf, L., Le, T.A., Wood, F. & Whiteson, S.. (2018). Deep Variational Reinforcement Learning for POMDPs. Proceedings of the 35th International Conference on Machine Learning, in PMLR 80:2117-2126
** Question **
I have several questions where I'm not sure I understand the paper correctly:
1.) Why can FP only predict the mean? For example, one could use a PixelCNN as decoder, which would allow to learn an entire distribution, not just the mean over images.
2.) The problem that CPC and CPC|Action is unable to predict objects if they don't influence the future trajectory doesn't seem surprising to me because whether an image is a positive or negative example can usually be determined by the background, the object is not necessary to do so. In other words, this is a problem of how the negative samples are chosen: If they were constructed using a simulator that shows the same background but without the objects, the belief would need to start encoding the presence of objects. Is this correct or am I missing something?
3.) Am I correct in thinking that CPC(|Action) would not be applicable to properly estimate the belief distribution in the presence of noise, i.e. for example when estimation the exact location based on sensors with Gaussian noise?
** Overall **
* Pros:
- Extensive, interesting evaluation
- Novel CPC|Action algorithm
* Cons:
- No theoretical analysis/justification for claims
- There are several subtleties that I am not sure are sufficiently discussed in the paper (see my questions above)