Unsupervised Video Object Segmentation for Deep Reinforcement Learning
We present a new technique for deep reinforcement learning that automatically detects moving objects and uses the relevant information for action selection. The detection of moving objects is done in an unsupervised way by exploiting structure from motion. Instead of directly learning a policy from raw images, the agent first learns to detect and segment moving objects by exploiting flow information in video sequences. The learned representation is then used to focus the policy of the agent on the moving objects. Over time, the agent identifies which objects are critical for decision making and gradually builds a policy based on relevant moving objects. This approach, which we call Motion-Oriented REinforcement Learning (MOREL), is demonstrated on a suite of Atari games where the ability to detect moving objects reduces the amount of interaction needed with the environment to obtain a good policy. Furthermore, the resulting policy is more interpretable than policies that directly map images to actions or values with a black box neural network. We can gain insight into the policy by inspecting the segmentation and motion of each object detected by the agent. This allows practitioners to confirm whether a policy is making decisions based on sensible information. Our code is available at https://github.com/vik-goel/MOREL.
************************* Summary: ************************* This paper describes MOREL,  an approach for automatically segmenting objects for reinforcement learning. In particular, this work uses SfM-Net [1], which learns to predict optical flow of a single image, to segment the objects in a state, and then uses this object-mask for reinforcement learning. MOREL is evaluated on all 59 Atari games, where it outperforms the baselines in several environments.   ************************* Quality: ************************* This approach appears to be sound. Additionally, it is evaluated within several environments. The paper does not have enough discussion about weaknesses of the approach. Additionally, while there is a clear advantage of using MOREL over the baseline approaches, it would be better to have more than three averaged results.  ************************* Clarity: ************************* This paper is well-written. I did not have any problems understanding the approach or motivation. I did not even find any typos. I do have a couple of comments, though. In the background, the policy is defined w.r.t. histories, but this is not a common way to formulate policies. It is unclear why it is defined in this way in the paper. Additionally, r is never defined.   ************************* Originality: ************************* The main contribution of this work is using SfM-Net to segment images for reinforcement learning. This is similar to feature extraction, and other similar approaches should have been discussed or compared to. In general, the related work lacks discussion of related feature construction and computer vision techniques. Atari is simple and the features could be constructed using very basic computer vision techniques. In fact, Bellemare et al. [2] list a many ways to extract features for Atari, for example by using background subtraction. Additionally, DARLA [3] uses Beta-VAE to learn disentangled features, such as objects, for reinforcement learning, and deep spatial auto-encoders [4] also learn features that encode information about objects.  ************************* Significance: ************************* This work introduces an interesting way to segment objects for reinforcement learning. Vijayanarasimhan et al [1] demonstrate that Sfm-net can work on realistic environments, but it would still be useful to also evaluate MOREL within less trivial environments, particularly because the backgrounds in Atari games are stationary. This would make it clear that it is beneficial to segment objects. For example, do the masks even need to be learned? Can the objects be detected by computing optical flow of the previous and current frames or background subtraction? Can pre-trained features be used?  Additionally, the paper described few changes to SfM-Net. It would have been nice to see results for the unchanged network.  The main changes were a different loss for reconstruction, flow regularization, and a curriculum. These changes are incremental, and so the segmentation results from Figure 3 seem more like result from running SfM-Net on Atari, rather than any contribution from this work. This statement could be argued against by demonstrating that the adjustments to SfM-Net yield different or better results than the unchanged network.  Nevertheless, the approach performs well against the baselines, and I do believe it could be a useful approach since it would likely make it easier to learn about salient objects.   ************************* Comments ************************* The results do not demonstrate that this representation is more interpretable. Detecting objects does not give any insight into the decisions being made by the policy. Using attention might change this. Line 234: I disagree that the segmentation is remarkable since the approach is only evaluated on Atari. Additionally, the model is self-supervised. It is given optical flow and thus predicts optical flow. So the result does not seem too surprising. Why should we assume that moving objects are important?  ************************* Typos ************************* None, well done!  ************************* References: ************************* [1] Vijayanarasimhan, Sudheendra, et al. "Sfm-net: Learning of structure and motion from video." arXiv preprint arXiv:1704.07804 (2017). [2] Bellemare, Marc G., et al. "The arcade learning environment: An evaluation platform for general agents." Journal of Artificial Intelligence Research 47 (2013): 253-279. [3] Higgins, Irina, et al. "Darla: Improving zero-shot transfer in reinforcement learning." arXiv preprint arXiv:1707.08475 (2017). [4] Finn, Chelsea, et al. "Deep spatial autoencoders for visuomotor learning." arXiv preprint arXiv:1509.06113 (2015).  Update: I have read the review's and author's response. While I appreciate the rigorous studies done in the atari environment, I still believe the paper would be greatly improved if it demonstrated the benefits of the approach in environments with more difficult segmentation. This could still be done in simulation. As such, my score remains the same.