Learning Physics Priors for Deep Reinforcement Learing
While model-based deep reinforcement learning (RL) holds great promise for sample efficiency and generalization, learning an accurate dynamics model is challenging and often requires substantial interactions with the environment. Further, a wide variety of domains have dynamics that share common foundations like the laws of physics, which are rarely exploited by these algorithms. Humans often acquire such physics priors that allow us to easily adapt to the dynamics of any environment. In this work, we propose an approach to learn such physics priors and incorporate them into an RL agent. Our method involves pre-training a frame predictor on raw videos and then using it to initialize the dynamics prediction model on a target task. Our prediction model, SpatialNet, is designed to implicitly capture localized physical phenomena and interactions. We show the value of incorporating this prior through empirical experiments on two different domains – a newly created PhysWorld and games from the Atari benchmark, outperforming competitive approaches and demonstrating effective transfer learning.
 Summary This paper propose to learning a dynamics model with future prediction in video and using it for reinforcement learning. The dynamics models is a variants of convolution LSTM and it is trained mean squared error in the future frame. The way of using dynamics model for reinforcement learning is similar to Weber et al., 2017, where K step prediction of the dynamics model is uses as an augmented input of the policy. Strength Training dynamic model to understand physic and using it for reinforcement learning is an interesting problem that worth exploring. This paper tackles this problem and demonstrated experimental setting based on physics games. Weakness The part for understanding dynamics model is very close to existing convolutional LSTM model (Xingjian et al., 2015), which is a popular baseline in video modelling community and how pretrained dynamics model is used for reinforcement learning is similar to Weber et al., 2017, but this paper does not provide comparison to any of these two baseline. Since the difference with these existing method is subtle, clear comparison with these method and difference in characteristic is essential to show the novelty of the paper. Overall comment This paper address the interesting problem of understanding dynamics for solving reinforcement learning, but the suggested method is not novel and comparison with existing close methods are not performed.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Learning Physics Priors for Deep Reinforcement Learing
While model-based deep reinforcement learning (RL) holds great promise for sample efficiency and generalization, learning an accurate dynamics model is challenging and often requires substantial interactions with the environment. Further, a wide variety of domains have dynamics that share common foundations like the laws of physics, which are rarely exploited by these algorithms. Humans often acquire such physics priors that allow us to easily adapt to the dynamics of any environment. In this work, we propose an approach to learn such physics priors and incorporate them into an RL agent. Our method involves pre-training a frame predictor on raw videos and then using it to initialize the dynamics prediction model on a target task. Our prediction model, SpatialNet, is designed to implicitly capture localized physical phenomena and interactions. We show the value of incorporating this prior through empirical experiments on two different domains – a newly created PhysWorld and games from the Atari benchmark, outperforming competitive approaches and demonstrating effective transfer learning.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 Summary This paper propose to learning a dynamics model with future prediction in video and using it for reinforcement learning. The dynamics models is a variants of convolution LSTM and it is trained mean squared error in the future frame. The way of using dynamics model for reinforcement learning is similar to Weber et al., 2017, where K step prediction of the dynamics model is uses as an augmented input of the policy. Strength Training dynamic model to understand physic and using it for reinforcement learning is an interesting problem that worth exploring. This paper tackles this problem and demonstrated experimental setting based on physics games. Weakness The part for understanding dynamics model is very close to existing convolutional LSTM model (Xingjian et al., 2015), which is a popular baseline in video modelling community and how pretrained dynamics model is used for reinforcement learning is similar to Weber et al., 2017, but this paper does not provide comparison to any of these two baseline. Since the difference with these existing method is subtle, clear comparison with these method and difference in characteristic is essential to show the novelty of the paper. Overall comment This paper address the interesting problem of understanding dynamics for solving reinforcement learning, but the suggested method is not novel and comparison with existing close methods are not performed.
assistant
 Summary This paper propose to learning a dynamics model with future prediction in video and using it for reinforcement learning. The dynamics models is a variants of convolution LSTM and it is trained mean squared error in the future frame. The way of using dynamics model for reinforcement learning is similar to Weber et al., 2017, where K step prediction of the dynamics model is uses as an augmented input of the policy. Strength Training dynamic model to understand physic and using it for reinforcement learning is an interesting problem that worth exploring. This paper tackles this problem and demonstrated experimental setting based on physics games. Weakness The part for understanding dynamics model is very close to existing convolutional LSTM model (Xingjian et al., 2015), which is a popular baseline in video modelling community and how pretrained dynamics model is used for reinforcement learning is similar to Weber et al., 2017, but this paper does not provide comparison to any of these two baseline. Since the difference with these existing method is subtle, clear comparison with these method and difference in characteristic is essential to show the novelty of the paper. Overall comment This paper address the interesting problem of understanding dynamics for solving reinforcement learning, but the suggested method is not novel and comparison with existing close methods are not performed.