Learning control policies in robotic tasks requires a large number of interactions due to small learning rates, bounds on the updates or unknown constraints. In contrast humans can infer protective and safe solutions after a single failure or unexpected observation. In order to reach similar performance, we developed a hierarchical Bayesian optimization algorithm that replicates the cognitive inference and memorization process for avoiding failures in motor control tasks. A Gaussian Process implements the modeling and the sampling of the acquisition function. This enables rapid learning with large learning rates while a mental replay phase ensures that policy regions that led to failures are inhibited during the sampling process. The features of the hierarchical Bayesian optimization method are evaluated in a simulated and physiological humanoid postural balancing task. We quantitatively compare the human learning performance to our learning approach by evaluating the deviations of the center of mass during training. Our results show that we can reproduce the efficient learning of human subjects in postural control tasks which provides a testable model for future physiological motor control tasks. In these postural control tasks, our method outperforms standard Bayesian Optimization in the number of interactions to solve the task, in the computational demands and in the frequency of observed failures.
After rebuttal:
Thank you to the authors for responding to my review.
1) The title of the conference is "... on Learning Representations". As I stated in the review ("no, e.g., neural networks are employed"), neural networks are an *example* of, but do not subsume, all representation learning methods. Therefore, I agree that papers that do not cover neural networks are welcome at the conference. However, as I stated in the review, my evaluation of the method proposed in the submission is that it does not concern representation learning ("The employed features in Table 3 are handcrafted"). I believe this evaluation is defensible, but of course the final evaluation is up to the chairs. However, I note that the authors did not respond directly to my evaluation that the method is not engaging in representation learning.
2-7) As the other reviewer notes, the paper lacks clarity in many places, and does not sufficiently discuss prior work, including in postural control (there is one citation in the references that is not mentioned in the main text), hierarchical Bayesian optimization within or without a Gaussian processes framework (https://scholar.google.com/scholar?hl=fr&as_sdt=0%2C5&q=hierarchical+bayesian+optimization&btnG=), or experience replay (https://scholar.google.com/scholar?hl=fr&as_sdt=0%2C5&q=replay+machine+learning&btnG=). Therefore, it is difficult to ascertain the research contribution.
As such, I stand by my evaluation that this submission is not ready for publication at ICLR.
===========================
Before rebuttal:
The submission presents a hierarchical Bayesian optimization (HiBO) approach to solving a postural control task expressed as a proportional-derivative (PD) controller.
Strengths:
- The HiBO approach outperforms the non-hierarchical BO approach on the task of postural control.
Weaknesses:
- The paper does not make use of representation learning (no, e.g., neural networks are employed) and is therefore out-of-place at ICLR. The employed features in Table 3 are handcrafted.
- The task (simulating human postural control) is not well-situated in the context of prior work using HiBO for robotics, so the contribution remains unclear.
- It is not clear why this problem should be formulated as contextual policy search (i.e., to what the context variable refers).
- Only one baseline (Bayesian optimization (BO)) is reported. This baseline corresponds to the ablation of the HiBO method (i.e., the omission of the context variable), and so does not represent, more broadly, an alternative approach.
- The concept of "mental replay" is briefly introduced, but no reference is made to prior work in imagined rollouts, and no ablation study on the impact of this component is performed.
Minor points:
- It is unclear why the problem setting should be labeled as "psychological" postural control.
- There are several missing references ("?") in the text.