Robust Reinforcement Learning aims to find the optimal policy with some extent of robustness to environmental dynamics. Existing learning algorithms usually enable the robustness though disturbing the current state or simulated environmental parameters in a heuristic way, which lack quantified robustness to the system dynamics (i.e. transition probability). To overcome this issue, we leverage Wasserstein distance to measure the disturbance to the reference transition kernel. With Wasserstein distance, we are able to connect transition kernel disturbance to the state disturbance, i.e. reduce an infinite-dimensional optimization problem to a finite-dimensional risk-aware problem. Through the derived risk-aware optimal Bellman equation, we show the existence of optimal robust policies, provide a sensitivity analysis for the perturbations, and then design a novel robust learning algorithm-Wasserstein Robust Advantage Actor-Critic algorithm (WRAAC). The effectiveness of the proposed algorithm is verified in the Cart-Pole environment.
This work aims to produce reinforcement learning methods that are ‘distributionally robust’. They approach this by assuming the transition function may vary between elements of the domain distribution and extend some recent results (Blanchet & Murthy, 2019) to give some theoretical results (contraction and optimal deterministic policy) and an actor-critic algorithm. This is an important research area and the work takes a very nice approach. However, the clarity of the work and the empirical results could both use some work.
There are a great many grammatical errors throughout the paper. The nature of the errors suggests a non-native speaker, which I really do not want to discourage, but this needs a proofread and edit. One exception, there is an actual typo just after assumption 1 “dfnied”, but actually better grammatically to just remove the word entirely.
In the intro, the citation of Mannor et al. 2004; 2007 for what is essentially a description of the gap between simulation to real transfer is a very strange choice. Is it possible this was a mistake?
The description leading up to the main result could be made much clearer. I went through this section, the main results, and the proofs in the appendix. Things look good, although I do feel like (as just said) it could be made much better/clearer. One snag I hit was in the proof of Lemma 1, everything is good except I don’t know why “u_1 \le u_2 + \gamma \| u_1 - u_2 \|_\infty” must hold, but I might have simply missed something obvious so please help me out.
The experimental results are very minimal and not very convincing. It is not immediately clear to me that Figures 1&2 actually show that the robustac algorithm is more robust than ac. The performance is worse than ac, which is completely understandable, but for most of the environments ac is still significantly better.
Lastly, a small point to notice that this paper bleeds into the 9th page, and considering the amount of contributions (largely down to the two theoretical results and a very small experiment) I do not think the extra space is warranted. This could be compressed and actually come out stronger as a result of being forced to be clearer and more concise.
Update:
Thank you for your response, especially the fix to the proof. I will keep my score, but do believe that further experiments and small adjustments to the writing will see a future version of this accepted.