Learning by shaking: Computing policy gradients by physical forward-propagation
Model-free and model-based reinforcement learning are two ends of a spectrum. Learning a good policy without a dynamic model can be prohibitively expensive. Learning the dynamic model of a system can reduce the cost of learning the policy, but it can also introduce bias if it is not accurate. We propose a middle ground where instead of the transition model, the sensitivity of the trajectories with respect to the perturbation (shaking) of the parameters is learned. This allows us to predict the local behavior of the physical system around a set of nominal policies without knowing the actual model. We assay our method on a custom-built physical robot in extensive experiments and show the feasibility of the approach in practice. We investigate potential challenges when applying our method to physical systems and propose solutions to each of them.
The paper proposes learning physical derivatives, the derivative of the trajectory distribution with respect to policy parameters. The proposed method estimates changes in trajectories at a particular theta by using finite differences, then fitting Gaussian Processes per timestep to generalize to new dtheta*s. The paper then proposes techniques to robustify the process against noise in the system. To deal with temporal noise, where trajectories are approximately equal up to a time shift, they simply estimate the optimal shift and use the shifted version to estimate. To address more complicated noise, they assume sensitivity of trajectories to noise is small relative to sensitivity to the parameters, and discretize the state space at a level such that trajectories that differ primarily due to inherent noise look the same at the discretized level, while perturbed policy parameters still lead to different trajectories. They then use the discretized trajectories to estimate the finite differences. Experiments illustrate how the learned predictions compared to actual resulting perturbations and illustrate resulting trajectories on certain toy domains and a physical robotic finger. All experiments are done with very low dimensional state and policy spaces (1-3 dimensions each). Without much more extensive experimental validation, this paper should be rejected. While I am not aware of any prior work on learning physical derivatives, the actual methods used are not novel in of themselves beyond being applied towards learning derivatives with respect to the policy. As such, the method should be of practical interest in order to be accepted. With limited experiments on a single very low dimensional domain and no comparisons against any alternative methods, there is little evidence demonstrating the actual effectiveness of the proposed method, especially on more complex domains and for downstream tasks. Suggested Experiments: - Stability in gradient estimation It seems like it could require huge amounts of samples to be able to estimate gradients at parameters where for which the system is not very stable, as states in at later timesteps can easily change in hard to predict ways as the dynamics are propogated through time. This should be an especially big issue if we do not already have good stable controllers close to the desired solution and needed to actually conduct exploration in parameter space to solve a taask. I would appreciate more extensive evaluation across multiple different (simulated) domains and assessing the effectiveness of gradient estimation along random parameters. - Dimensionality of policy parameters and state spaces The current experiments only involve very small parameter spaces. It would be important to see how well using finite differences and GP regression scales with a higher dimensional search space, which can be demonstrated on varying dimensionalities of an LQR system for example. It would also be important to see how gradient estimation scales with high dimensional state spaces even with a small parameter space (like in the PD controller experiment in the paper). - Direct Comparison against learning dynamics models Using the same data, compare (with same metrics as in table 1) physical derivatives estimated with the proposed approach against learning GP dynamics models and rolling out the perturbed policy with the learned model. Without a direct comparison against learning dynamics models and understanding what situations learning physical derivatives provides better estimates, it is unclear when or why one would prefer to learn physical derivatives in this way compared to a model based approach. - Quantitative results measuring costs of learned controllers Despite the name of the paper and a description of how to compute a policy gradient via physical derivatives, there are no experiments involving such policy gradient updates as far as I can tell. While one advantage of the method (as well as model based approaches) is the ability to learn in unsupervised manner, it would be extremely helpful to validate how well the physical derivatives are estimated in terms of how useful they are for a downstream task, such as optimizing a controller for a cost function. Right now, experimental results lack any comparisons to other methods or any other way to assess the effectiveness of estimating physical derivatives. A comparison against regular RL policy gradient methods (or other model free algorithms) and model based RL would give an idea as to whether the physical derivatives learned are actually useful. Other questions and comments: - Table 1: is this evaluating the accuracy of the physical derivatives on the shaking data that it was used for learning, or on a validation set? If on a validation set, would the validation perturbations be drawn from the same distribution as the training set? - The zero shot planning experiment in section 4.4 seems very contrived. It does not seem like a useful task to adjust the parameters of the PD controller in order to reach a state that isn*t the target. The figures illustrating trajectories are also not very convincing and unclear. Two points are labelled source state and target state, but it is not clear which is the intermediate state it is supposed to reach. In any case, most of the trajectories seem to vastly overshoot the target? final state, and it is hard to assess how close the trajectories end up being to the intended states from a 2d representation of a 3d space. Quantitative results would perhaps have been more useful in illustrating the effectiveness of using physical derivatives. - What is the purpose of figure 4? It does not appear to be referenced in the text and it is not clear what is being shown. Other notes not part of decision: Paper exceeds the 8 page recommended length Lots of small typos in the text
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Learning by shaking: Computing policy gradients by physical forward-propagation
Model-free and model-based reinforcement learning are two ends of a spectrum. Learning a good policy without a dynamic model can be prohibitively expensive. Learning the dynamic model of a system can reduce the cost of learning the policy, but it can also introduce bias if it is not accurate. We propose a middle ground where instead of the transition model, the sensitivity of the trajectories with respect to the perturbation (shaking) of the parameters is learned. This allows us to predict the local behavior of the physical system around a set of nominal policies without knowing the actual model. We assay our method on a custom-built physical robot in extensive experiments and show the feasibility of the approach in practice. We investigate potential challenges when applying our method to physical systems and propose solutions to each of them.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



The paper proposes learning physical derivatives, the derivative of the trajectory distribution with respect to policy parameters. The proposed method estimates changes in trajectories at a particular theta by using finite differences, then fitting Gaussian Processes per timestep to generalize to new dtheta*s. The paper then proposes techniques to robustify the process against noise in the system. To deal with temporal noise, where trajectories are approximately equal up to a time shift, they simply estimate the optimal shift and use the shifted version to estimate. To address more complicated noise, they assume sensitivity of trajectories to noise is small relative to sensitivity to the parameters, and discretize the state space at a level such that trajectories that differ primarily due to inherent noise look the same at the discretized level, while perturbed policy parameters still lead to different trajectories. They then use the discretized trajectories to estimate the finite differences. Experiments illustrate how the learned predictions compared to actual resulting perturbations and illustrate resulting trajectories on certain toy domains and a physical robotic finger. All experiments are done with very low dimensional state and policy spaces (1-3 dimensions each). Without much more extensive experimental validation, this paper should be rejected. While I am not aware of any prior work on learning physical derivatives, the actual methods used are not novel in of themselves beyond being applied towards learning derivatives with respect to the policy. As such, the method should be of practical interest in order to be accepted. With limited experiments on a single very low dimensional domain and no comparisons against any alternative methods, there is little evidence demonstrating the actual effectiveness of the proposed method, especially on more complex domains and for downstream tasks. Suggested Experiments: - Stability in gradient estimation It seems like it could require huge amounts of samples to be able to estimate gradients at parameters where for which the system is not very stable, as states in at later timesteps can easily change in hard to predict ways as the dynamics are propogated through time. This should be an especially big issue if we do not already have good stable controllers close to the desired solution and needed to actually conduct exploration in parameter space to solve a taask. I would appreciate more extensive evaluation across multiple different (simulated) domains and assessing the effectiveness of gradient estimation along random parameters. - Dimensionality of policy parameters and state spaces The current experiments only involve very small parameter spaces. It would be important to see how well using finite differences and GP regression scales with a higher dimensional search space, which can be demonstrated on varying dimensionalities of an LQR system for example. It would also be important to see how gradient estimation scales with high dimensional state spaces even with a small parameter space (like in the PD controller experiment in the paper). - Direct Comparison against learning dynamics models Using the same data, compare (with same metrics as in table 1) physical derivatives estimated with the proposed approach against learning GP dynamics models and rolling out the perturbed policy with the learned model. Without a direct comparison against learning dynamics models and understanding what situations learning physical derivatives provides better estimates, it is unclear when or why one would prefer to learn physical derivatives in this way compared to a model based approach. - Quantitative results measuring costs of learned controllers Despite the name of the paper and a description of how to compute a policy gradient via physical derivatives, there are no experiments involving such policy gradient updates as far as I can tell. While one advantage of the method (as well as model based approaches) is the ability to learn in unsupervised manner, it would be extremely helpful to validate how well the physical derivatives are estimated in terms of how useful they are for a downstream task, such as optimizing a controller for a cost function. Right now, experimental results lack any comparisons to other methods or any other way to assess the effectiveness of estimating physical derivatives. A comparison against regular RL policy gradient methods (or other model free algorithms) and model based RL would give an idea as to whether the physical derivatives learned are actually useful. Other questions and comments: - Table 1: is this evaluating the accuracy of the physical derivatives on the shaking data that it was used for learning, or on a validation set? If on a validation set, would the validation perturbations be drawn from the same distribution as the training set? - The zero shot planning experiment in section 4.4 seems very contrived. It does not seem like a useful task to adjust the parameters of the PD controller in order to reach a state that isn*t the target. The figures illustrating trajectories are also not very convincing and unclear. Two points are labelled source state and target state, but it is not clear which is the intermediate state it is supposed to reach. In any case, most of the trajectories seem to vastly overshoot the target? final state, and it is hard to assess how close the trajectories end up being to the intended states from a 2d representation of a 3d space. Quantitative results would perhaps have been more useful in illustrating the effectiveness of using physical derivatives. - What is the purpose of figure 4? It does not appear to be referenced in the text and it is not clear what is being shown. Other notes not part of decision: Paper exceeds the 8 page recommended length Lots of small typos in the text
assistant
The paper proposes learning physical derivatives, the derivative of the trajectory distribution with respect to policy parameters. The proposed method estimates changes in trajectories at a particular theta by using finite differences, then fitting Gaussian Processes per timestep to generalize to new dtheta*s. The paper then proposes techniques to robustify the process against noise in the system. To deal with temporal noise, where trajectories are approximately equal up to a time shift, they simply estimate the optimal shift and use the shifted version to estimate. To address more complicated noise, they assume sensitivity of trajectories to noise is small relative to sensitivity to the parameters, and discretize the state space at a level such that trajectories that differ primarily due to inherent noise look the same at the discretized level, while perturbed policy parameters still lead to different trajectories. They then use the discretized trajectories to estimate the finite differences. Experiments illustrate how the learned predictions compared to actual resulting perturbations and illustrate resulting trajectories on certain toy domains and a physical robotic finger. All experiments are done with very low dimensional state and policy spaces (1-3 dimensions each). Without much more extensive experimental validation, this paper should be rejected. While I am not aware of any prior work on learning physical derivatives, the actual methods used are not novel in of themselves beyond being applied towards learning derivatives with respect to the policy. As such, the method should be of practical interest in order to be accepted. With limited experiments on a single very low dimensional domain and no comparisons against any alternative methods, there is little evidence demonstrating the actual effectiveness of the proposed method, especially on more complex domains and for downstream tasks. Suggested Experiments: - Stability in gradient estimation It seems like it could require huge amounts of samples to be able to estimate gradients at parameters where for which the system is not very stable, as states in at later timesteps can easily change in hard to predict ways as the dynamics are propogated through time. This should be an especially big issue if we do not already have good stable controllers close to the desired solution and needed to actually conduct exploration in parameter space to solve a taask. I would appreciate more extensive evaluation across multiple different (simulated) domains and assessing the effectiveness of gradient estimation along random parameters. - Dimensionality of policy parameters and state spaces The current experiments only involve very small parameter spaces. It would be important to see how well using finite differences and GP regression scales with a higher dimensional search space, which can be demonstrated on varying dimensionalities of an LQR system for example. It would also be important to see how gradient estimation scales with high dimensional state spaces even with a small parameter space (like in the PD controller experiment in the paper). - Direct Comparison against learning dynamics models Using the same data, compare (with same metrics as in table 1) physical derivatives estimated with the proposed approach against learning GP dynamics models and rolling out the perturbed policy with the learned model. Without a direct comparison against learning dynamics models and understanding what situations learning physical derivatives provides better estimates, it is unclear when or why one would prefer to learn physical derivatives in this way compared to a model based approach. - Quantitative results measuring costs of learned controllers Despite the name of the paper and a description of how to compute a policy gradient via physical derivatives, there are no experiments involving such policy gradient updates as far as I can tell. While one advantage of the method (as well as model based approaches) is the ability to learn in unsupervised manner, it would be extremely helpful to validate how well the physical derivatives are estimated in terms of how useful they are for a downstream task, such as optimizing a controller for a cost function. Right now, experimental results lack any comparisons to other methods or any other way to assess the effectiveness of estimating physical derivatives. A comparison against regular RL policy gradient methods (or other model free algorithms) and model based RL would give an idea as to whether the physical derivatives learned are actually useful. Other questions and comments: - Table 1: is this evaluating the accuracy of the physical derivatives on the shaking data that it was used for learning, or on a validation set? If on a validation set, would the validation perturbations be drawn from the same distribution as the training set? - The zero shot planning experiment in section 4.4 seems very contrived. It does not seem like a useful task to adjust the parameters of the PD controller in order to reach a state that isn*t the target. The figures illustrating trajectories are also not very convincing and unclear. Two points are labelled source state and target state, but it is not clear which is the intermediate state it is supposed to reach. In any case, most of the trajectories seem to vastly overshoot the target? final state, and it is hard to assess how close the trajectories end up being to the intended states from a 2d representation of a 3d space. Quantitative results would perhaps have been more useful in illustrating the effectiveness of using physical derivatives. - What is the purpose of figure 4? It does not appear to be referenced in the text and it is not clear what is being shown. Other notes not part of decision: Paper exceeds the 8 page recommended length Lots of small typos in the text