Reinforcement learning (RL) has led to increasingly complex looking behavior in recent years. However, such complexity can be misleading and hides over-fitting. We find that visual representations may be a useful metric of complexity, and both correlates well objective optimization and causally effects reward optimization. We then propose curious representation learning (CRL) which allows us to use better visual representation learning algorithms to correspondingly increase visual representation in policy through an intrinsic objective on both simulated environments and transfer to real images. Finally, we show better visual representations induced by CRL allows us to obtain better performance on Atari without any reward than other curiosity objectives.
This paper presents an empirical study of using error reduction as a curiosity measure. The authors consider an auto-encoder model, a colorization model and RND as intrinsic motivation signals. I find the write up very unclear and have trouble understanding what the claims are and how they are backed up. 
Major points:
* About the claims as stated on page 2: 1) The first claim I don't understand, I think what is meant is on navigation tasks they find "their measure of representation learning" (proposed in kolesnikov 2019) seems to correlate well with reward optimization. In 3.1 to back this claim they claim to test disentanglement but seem to test classification. I see no reason to not put that part in 4.1. 2) The authors claim to propose a new method: it can't be a separate representation network to derive rewards because that is what burda et al. and many many others do, it can't be the minimax formulation because that has been known for a while (e.g. predictability minimization schmidhuber) so I am not sure what the claim is about. What exactly is novel about the model. 3) CRL seems at best to outperform baselines on beamrider, qbert and riverraid but the results are impossible to assess. We don't know what the x axis is in figure 7 (is it frames, with or without repeats, is updates etc). Pong -12 is far from learnt for instance and its one of the easiest games. 
* suprisal objective and modeling objectives are very high level concepts that we can talk about in the introduction and conclusion but much more precise terms need to be employed in the model exposition. The readers need to know what sort of properties they should have ideally easily identify examples (without having to read 2 papers and a large survey). I would start with the minimax formula and then explain what is considered, how the different intrinsic rewards are added, if they are normalized, how they are weighted etc. etc.
* the details about failed experiments, environments and architecture should IMO be relegated to experiments
* It should be very clear early on that the model is separate from the representation. 
* auto-encoders are a large family of models and it is not clear from the paper which exact model is meant by the authors. Also, citing Bengio 2013 is NOT a valid citation for auto-encoders. The right citation depends on the model you apply please use that!