Towards Generalization and Simplicity in Continuous Control
This work shows that policies with simple linear and RBF parameterizations can be trained to solve a variety of widely studied continuous control tasks, including the gym-v1 benchmarks. The performance of these trained policies are competitive with state of the art results, obtained with more elaborate parameterizations such as fully connected neural networks. Furthermore, the standard training and testing scenarios for these tasks are shown to be very limited and prone to over-fitting, thus giving rise to only trajectory-centric policies. Training with a diverse initial state distribution induces more global policies with better generalization. This allows for interactive control scenarios where the system recovers from large on-line perturbations; as shown in the supplementary video.
The paper evaluates natural policy gradient algorithm with simple linear policies on a wide range of “challenging” problems from OpenAI MuJoco environment, and shows that these shallow policy networks can learn effective policies in most domains, sometimes faster than NN policies. It further explores learning robust and more global policies by modifying existing domains, e.g. adding many perturbations and diverse initial states.
The first part of the paper, while not proposing new approaches, offers interesting insights into the performance of linear policies, given plethora of prior work on applying NN policies as default on these problems. This part can be further strengthened by doing ablation study on the RL optimizer. Specifically, GAE, sigma vs alpha in Eq. 5, and small trajectory batch vs large trajectory batch (SGD vs batch opt). In particular, it is nice to confirm if there are subtle differences from prior NPG implementation with linear policies that made the results in this paper significantly better (e.g. in TRPO [Schulman et. al., 2015], one of the major differences was sigma vs alpha). 
It could be unsurprising that a global linear policy may perform very well in narrow state space. The second part of the paper tries to address such concern by modifying the environments such that the policies must learn globally robust policies to succeed. The paper then again showed that the RBF/linear policies could learn effective recover policies. It would be great to include corresponding learning curves for these in Section 4 beyond Figure 4, comparing NN, linear and RBF. Are there figures available? 
Additional comment:
- A nice follow-up experiment to show is if linear policies can solve control problems from vision, with some generic pre-trained convolutional filters (e.g. ImageNet). This would further show that existing domains are very simple and do not require deep end-to-end optimization.