We introduce a new approach for comparing reinforcement learning policies, using Wasserstein distances (WDs) in a newly defined latent behavioral space. We show that by utilizing the dual formulation of the WD, we can learn score functions over trajectories that can be in turn used to lead policy optimization towards (or away from) (un)desired behaviors. Combined with smoothed WDs, the dual formulation allows us to devise efficient algorithms that take stochastic gradient descent steps through WD regularizers. We incorporate these regularizers into two novel on-policy algorithms, Behavior-Guided Policy Gradient and Behavior-Guided Evolution Strategies, which we demonstrate can outperform existing methods in a variety of challenging environments. We also provide an open source demo 1 .
Summary
This paper proposes using Wasserstein Distances to measure the difference between higher-level functions of policies, which this paper terms as "behaviors". For example, one such behavior could be the distribution over final states given the policy, or the distribution over returns given policy. Through the lens of these behavioral embeddings, this paper recovers a few important special cases that are well-known in the literature including WD-based TRPO and distributional RL. This paper shows that the dual formulation of the Wasserstein Distance gives the ability to score individual policies based on a given "behavioral mapping".
Review
The idea of generalizing the trust-region of TRPO by using the WD measure and behavior maps is intriguing. The paper introduces many choices of behavior maps that could lead towards interesting algorithms that merit additional study. I think the connections between the proposed family of algorithms and WD-based TRPO and distributional RL is highly motivating.
There are, however, a several key issues with the empirical study of the proposed methods that make it challenging to assess the value of introducing another partially understood deep policy gradient algorithm. The results show only behavioral studies of the algorithm, not investigating the effects of the WD based regularizer or the effects of the choice of behavioral map. The results are also significantly limited in their statistical significance, making distinguishing between algorithms difficult in most cases. And the comparison of wall-clock time is particularly difficult to assess due to the uncountably many possible sources of noise when comparing wall-clock time of highly complex algorithms. In the following paragraphs, I will expand upon each of these points.
One of the key contributions of this paper is the ability to define regularizers based on the definition of the behavior map. The paper introduces many such behavior maps, those over state visitation, actions, and returns; however, the paper does not investigate the effects this novel choice has on the results. It is unclear if introducing each of these behavior maps leads towards different results or if they each induce roughly the same final performance of the agent. It would be highly valuable for me to see a more careful study of the effect of choosing each of these behavior maps on a single simple environment, clarifying that this formulation leads to a family of useful algorithms. As it stands, BGPG may only beat TRPO on these domains because it had more meta-parameters to choose between (BGPG introduces many new meta-parameters: choice of behavior map, kernel for produce RKHS, the meta-parameters of that kernel, the meta-parameters of the behavior map like trajectory length, the entropy regularization term in the WD, etc.). Without a careful study, or intuitive explanation of any of these parameter choices, it is unclear if BGPG won simply through overfitting to the problem.
The statistical significance of the proposed algorithm is impossible to assess in the given form. The comparisons are made using only five random seeds and the standard error bars are frequently quite large. I refer to Henderson et al. 2017 for further explanation as to why five random seeds is simply too few to provide a meaningful comparison between algorithms. Instead, I'll discuss a few of the results in particular. In figure 3a, I notice that the proposed method has high variance until it plateaus around -300 reward; why? In each of the remaining plots of Figure 3, I notice that the propose method is significantly higher variance than any of its competitors. This greatly leads me to suspect that the proposed method would have much lower average performance if studied across a greater number of random seeds. In the walker domain (Figure 3d), why do BGPG and TRPO both plateau at the same point for many timesteps, then eventually BGPG starts improving again? From Figure 5, the paper somewhat misleadingly states that "BGES is the only method that drives the agent to the goal in both settings." However Figure 5 on the left clearly shows that BGES and NSR-ES have indistinguishable performance, and the error bars indicate that neither significantly outperform NoisyNet-TRPO. In fact, due to the high variance of BGES it is unclear if it would drive the agent towards the goal on average if run over more random seeds. On the right, NoisyNet-TRPO noticeably outperforms BGES and is significantly lower variance. The language used to describe Figure 6 is strong, stating that Figure 6 "proves that the benefits come here not just from introducing the regularizer, but from its particular form." However, I tend to disagree that Figure 6 reliably proves anything. Although it appears that BGES is significantly outperforming other methods, the number of meta-parameters over which it gets to optimize makes it difficult to say if the performance gain is from overfitting to the problem or the form of the regularizer. Additionally, good performance demonstrated across a single problem is hardly proof especially using only five random seeds (Henderson et al. 2017 Figure 5).
Although this plays comparatively little role in my scoring of the paper, I feel it is necessary to discuss briefly. Because it is impossible to determine the source of the speed differences between the particle approximator and BGPG, the results in Figure 4 are extremely difficult to interpret. It simply could be that BGPG uses slightly more optimized code than the particle approximator, or it could be that there is in fact a significant performance difference between the algorithms. The only true way to discuss performance differences between algorithms generally is through computational complexity for these reasons. In the case that these statistical algorithms share the same computational complexity, then further analysis including convergence rates would be able to shed light on the speed difference. However, wall-clock time vs. performance has so many confounding factors, it is a fairly meaningless unit of measure. For an empirical investigation of speed, I would suggest giving each algorithm similar number of learning steps or similar number of updates to their weights and compare performance in this way.
Additional Comments (do not affect score)
I'm curious if this work could be extended to the off-policy case. It does seem like a minor disadvantage that distributional RL is well-defined in both the on-policy and off-policy cases, but the proposed family of methods is not. From my reading of the paper, it appears that there is little preventing this extension. Is this true?
--------
Edit after reading other reviews and rebuttals.
I appreciate the response noting that the meta-parameters of this algorithm were not tuned. I still feel that understanding the sensitivity to these parameters is important to understand how the choice of these parameters impacts the performance. While it is certainly likely that extensive tuning of these parameters could yield even more improved results, it is also quite possible that the chosen parameters have biased the results. If a more intuitive discussion of the choice of meta-parameters could be included in the paper, I would be more willing to concede this point.
I recognize that 3 or 5 random seeds are a standard in the deep RL community, but I think Henderson et al. 2017 make this point better than I could: this is not a standard that we should hold to. I think that using small domains and smaller networks would allow running across more random seeds and providing a much more careful scientific study. This paper does not convincingly show state of the art performance, a metric that is nearly impossibly to define in a field that moves as quickly as ours, so should not strive to follow the same demonstration study design that SOTA papers follow. A paper with strong theoretical motivations (such as this) should compliment with strong empirical understanding of how that theory translates to practice. Instead, having strong theoretical motivations backed by a demonstration that the algorithm still works is much less convincing (to me). I think a more convincing set of experiments that match well with the intended purpose of the paper (theoretical contribution of behavioral embeddings) would be an investigation to _how_ the optimization affects the learned policies or representations. Instead of performance benchmarks, this would provide understanding for the various effects of the new framework.
After the rebuttal phase, I now have an additional new concern about the clarity of the writing and the completeness of the empirical details. Many of the results do not mention most of the important details necessary to replicate or even interpret the results. I had assumed that details were consistent across figures (e.g. figure 3 uses 5 random seeds, so I assumed the same to be true for all other figures that did not specify), but the author response made clear that this is not true. In additional, little to no details are provided in the paper referring to the choices of meta-parameters for any algorithm or the methodology used to chose these meta-parameters. After these details were given in the rebuttal, my concerns were greatly alleviated, but not completely removed. It is unclear if the authors intend to update the paper before the final version, but based on the language of the rebuttal (e.g. indicating it is purely a misunderstanding of the paper), I am forced to assume that the paper will not include these important details.
To summarize. I think the theoretical underpinnings of these paper are extremely motivating and interesting. I do not want to see these innovations lost. However, the current state of the empirical section of the paper leaves too much open for me to be able to recommend an accept at this time. I do not believe a paper should solve everything in one pass (e.g. careful parameter studies, SOTA demonstrations, real-world demonstrations, etc.), but I do not believe this paper demonstrates any one of these convincingly.