Imitation Learning is the task of mimicking the behavior of an expert player in a Reinforcement Learning (RL) Environment to enhance the training of a fresh agent (called novice) beginning from scratch. Most of the Reinforcement Learning environments are stochastic in nature, i.e., the state sequences that an agent may encounter usually follow a Markov Decision Process (MDP). This makes the task of mimicking difficult as it is very unlikely that a new agent may encounter same or similar state sequences as an expert. Prior research in Imitation Learning proposes various ways to learn a mapping between the states encountered and the respective actions taken by the expert while mostly being agnostic to the order in which these were performed. Most of these methods need considerable number of states-action pairs to achieve good results. We propose a simple alternative to Imitation Learning by appending the novice's action space with the frequent short action sequences that the expert has taken. This simple modification, surprisingly improves the exploration and significantly outperforms alternative approaches like Dataset Aggregation. We experiment with several popular Atari games and show significant and consistent growth in the score that the new agents achieve using just a few expert action sequences.
[Summary]
This paper presents an interesting idea that to append the agent's action space with the expert's most frequent action pairs, by which the agent can perform better exploration as to achieve the same performance in a shorter time. The authors show performance gain by comparing their method with two baselines - Dagger and InfoGAIL.
[Stengths]
The proposed method is simple yet effective, and I really like the analogy to mini-moves in sports as per the motivation section.
[Concerns]
- How to choose the number and length of the action sequences?
The authors empirically add the same number of expert's action sequences as the basic ones and select the length k as 2. However, no ablation studies are performed to demonstrate the sensitivity of the selected hyperparameters. Although the authors claim that "we limit the size of meta-actions k to 2 because large action spaces may lead to poor convergence", a more systematic evaluation is needed. How will the performance change if we add more and longer action sequences? When will the performance reach a plateau? How does it vary between different environments?
- Analysis of the selected action sequences.
It might be better to add more analysis of the selected action sequences. What are the most frequent action pairs? How does it differ from game to game? What if the action pairs are selected in a random fashion?
- Justification of the motivation
The major motivation of the method is to release the burden of memory overheads. However, no quantitative evaluations are provided as to justify the claim. Considering that the input images are resized to 84x84, storing them should not be particularly expensive.
- The choice of baseline.
InfoGAIL (Li et al., 2017) is proposed to identify the latent structures in the expert's demonstration, hence it is not clear to me how it suits the tasks in the paper. The paper also lacks details describing how they implemented the baselines, e.g. beta in Dagger and the length of the latent vector in InfoGAIL.
- The authors only show experiments in Atari games, where the action space is discrete. It would be interesting to see if the idea can generalize to continuous action space. Is it possible to cluster the expert action sequences and form some basis for the agent to select?
- Typos
{LRR, RLR/RRL} --> {LRR, RLR, RRL}
sclability --> scalability
we don't need train a ... --> we don't need to train a ...
Atmost --> At most
[Recommendation]
The idea presents in the paper is simple yet seemingly effective. However, the paper lacks a proper evaluation of the proposed method, and I don't think this paper is ready with the current set of experiments. I will decide my final rating based on the authors' response to the above concerns.