Policy Optimization via Importance Sampling
Policy optimization is an effective reinforcement learning approach to solve continuous control tasks. Recent achievements have shown that alternating online and offline optimization is a successful choice for efficient trajectory reuse. However, deciding when to stop optimizing and collect new trajectories is non-trivial, as it requires to account for the variance of the objective function estimate. In this paper, we propose a novel, model-free, policy search algorithm, POIS, applicable in both action-based and parameter-based settings. We first derive a high-confidence bound for importance sampling estimation; then we define a surrogate objective function, which is optimized offline whenever a new batch of trajectories is collected. Finally, the algorithm is tested on a selection of continuous control tasks, with both linear and deep policies, and compared with state-of-the-art policy optimization methods.
[UPDATE: I have read the author responses and I am satisfied with them]  This paper proposes a novel importance sampling policy search method which accounts for the problem of large variance (low effective sample size) if one just does naive IS and the target distribution differs a lot from the behavior distribution. The authors take a more conservative approach, deriving an update rule based on natural gradient and considering a surrogate which is derived analytically. This is in essence the principle of most of the state-of-the-art algorithms for policy search. They develop both control-based and parameter-based formulations and provide a detailed comparison with some state-of-the-art policy optimization methods.  This is a very well written paper and a solid piece of work. There are several nice contributions in this paper: from a theoretical side, the authors bring to the policy optimization field a set of useful tools from sampling and stochastic calculus which I don't think they are well established in the community. Moreover, they extensively review a vast piece of existing research done in the area. In addition, the proposed algorithms have comparable performance to the state-of-the-art methods.  I have the following comments/questions to the authors:  1- line 185: "unacceptable limitations (...) target variance larger than the behavioral one" and line 109: "we require \sigma_P < 2\sigma_Q".     -1st question: Is this not a contradiction?     -2nd why would one use a target variance larger than the behavioral one? Usually, the behavioral one is more explorative (larger variance) and not the other way around.     -3rd can you relate this with the experiments?  2- While it seems that authors consider general stochastic systems (line 100), they only focus on deterministic systems. In that sense, for example, (line 237) is misleading, since for deterministic systems the probability of a trajectory can be calculated without knowing the model. Isn't it? I strongly suggest to state from the beginning whether they consider deterministic dynamical systems or not.  3 - The numerical experiments are a bit disconnected from the theoretical parts of the paper. For example, given the strong emphasis that is made on the Effective Sample Size, I was expecting some numerical demonstration that showed the increment in the ESS during optimization. Since the bound is directly related with the ESS, why not reporting that as well? It is nice to see performance curves that compare other algorithms, but sometimes is even better to have a theory section well integrated with the empirical one.  4- I would remove the offline/online narrative, which is confusing. For example, in the abstract it looks like the algorithm is just optimizing offline, but in reality it does not.  Other:  - Eq(1): outer integral should be over $\theta$? - line 104: integral again - line 155: "access to set" -> "access to a set" - line 218: swap small o with the plus sign