Reinforcement learning (RL) often requires large numbers of trials to solve a single specific task. This is in sharp contrast to human and animal learning: humans and animals can use past experience to acquire an understanding about the world, which they can then use to perform new tasks with minimal additional learning. In this work, we study how an unsupervised exploration phase can be used to build up such prior knowledge, which can then be utilized in a second phase to perform new tasks, either directly without any additional exploration, or through minimal fine-tuning. A critical question with this approach is: what kind of knowledge should be transferred from the unsupervised phase to the goal-directed phase? We argue that model-based RL offers an appealing solution. By transferring models, which are task-agnostic, we can perform new tasks without any additional learning at all. However, this relies on having a suitable exploration method during unsupervised training, and a model-based RL method that can effectively utilize modern high-capacity parametric function classes, such as deep neural networks. We show that both challenges can be addressed by representing model-uncertainty, which can both guide exploration in the unsupervised phase and ensure that the errors in the model are not exploited by the planner in the goal-directed phase. We illustrate, on simple simulated benchmark tasks, that our method can perform various goal-directed skills on the first attempt, and can improve further with finetuning, exceeding the performance of alternative exploration methods.
The authors address the problem of how to use unsupervised exploration in a first phase of reinforcement learning to gather knowledge that can be transferred to new tasks to improve performance in a second task when specific reward functions are available. The authors proposed a model-based approach which uses deep neural networks as a model for the environment. The model is PETS (probabilistic ensembles with trajectory sampling), an ensemble of neural networks whose outputs parametrize predictive distributions for the next state as a function of the current state and the action applied. To collect data during the unsupervised exploration phase, they use a metric of model uncertainty computed as follows: the average over all the particles assigned to each bootstrap is computed and the variance over these computed means is the
metric of uncertainty. The authors validate their method on the HalfCheetah OpenAI gym environment where they consider 4 different tasks related to running forward, backward, tumbling forward and tumbling backward. The results obtained show that they outperform random and count based exploration approaches.
Quality:
I am concerned about the quality of the experimental evaluation of the method. The authors only consider a single environment for their experiments and artificially construct 4 relatively similar tasks. I believe this is insufficient to quantify the usefulness of the proposed method.
Clarity:
The paper is clearly written and easy to read.
Novelty:
The proposed approach seems incremental and lacks novelty. The described method for model-based exploration consists in looking at the mean of the prediction of each neural network in the ensemble and then computing the empirical average. This approach has been used before for active learning with neural networks ensembles:
Krogh, Anders, and Jesper Vedelsby. "Neural network ensembles, cross validation, and active learning." Advances in neural information processing systems. 1995.
The used model, PETS, is also not novel and the proposed methodology for having first an unsupervised learning phase and then a new specific learning task is also not very innovative.
Significance:
Given the lack of a rigorous evaluation framework and the lack of novelty of the proposed methods, I believe the significance of the contribution is very low.