SIMILE: Introducing Sequential Information towards More Effective Imitation Learning
Reinforcement learning (RL) is a metaheuristic aiming at teaching an agent to interact with an environment and maximizing the reward in a complex task. RL algorithms often encounter the difficulty in defining a reward function in a sparse solution space. Imitation learning (IL) deals with this issue by providing a few expert demonstrations, and then either mimicking the expert*s behavior (behavioral cloning, BC) or recovering the reward function by assuming the optimality of the expert (inverse reinforcement learning, IRL). Conventional IL approaches formulate the agent policy by mapping one single state to a distribution over actions, which did not consider sequential information. This strategy can be less accurate especially in IL, a weakly supervised learning environment, especially when the number of expert demonstrations is limited. This paper presents an effective approach named Sequential IMItation LEarning (SIMILE). The core idea is to introduce sequential information, so that an agent can refer to both the current state and past state-action pairs to make a decision. We formulate our approach into a recurrent model, and instantiate it using LSTM so as to fuse both long-term and short-term information. SIMILE is a generalized IL framework which is easily applied to BL and IRL, two major types of IL algorithms. Experiments are performed on several robot controlling tasks in OpenAI Gym. SIMILE not only achieves performance gain over the baseline approaches, but also enjoys the benefit of faster convergence and better stability of testing performance. These advantages verify a higher learning efficiency of SIMILE, and implies its potential applications in real-world scenarios, i.e., when the agent-environment interaction is more difficult and/or expensive.
 This paper introduces the use of sequential information (state-action pairs) for enhancing imitation learning, and using recurrent networks (LSTM) in that process. The authors motivate this by pointing out that while the state information, if Markovian, should contain all information necessary for decision making, with incomplete learners redundant information in the sequential state-action information leading to the current state can be helpful, citing some concrete examples. After describing a number of variants of this idea, in the context of IRL, BC, etc., the authors conduct a systematic empirical evaluation to assess the effectiveness of the proposal, over the baselines, using a number of RL benchmark problems. The results are favorable and convincingly show that the proposed sequential enhancement can bring significant improvement in terms of attained rewards, convergence speed and stability in many of the tested cases. One suggestion I have is that it would be interesting to investigate into the question of how the addition of sequential information adds value is related to the validity of Markovian assumption in each of the problem being considered. It is a good empirical paper demonstrating the practical use of an idea that is simple but reasonable, and in a way that is substantiated using proper cutting edge framework and baselines.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
SIMILE: Introducing Sequential Information towards More Effective Imitation Learning
Reinforcement learning (RL) is a metaheuristic aiming at teaching an agent to interact with an environment and maximizing the reward in a complex task. RL algorithms often encounter the difficulty in defining a reward function in a sparse solution space. Imitation learning (IL) deals with this issue by providing a few expert demonstrations, and then either mimicking the expert*s behavior (behavioral cloning, BC) or recovering the reward function by assuming the optimality of the expert (inverse reinforcement learning, IRL). Conventional IL approaches formulate the agent policy by mapping one single state to a distribution over actions, which did not consider sequential information. This strategy can be less accurate especially in IL, a weakly supervised learning environment, especially when the number of expert demonstrations is limited. This paper presents an effective approach named Sequential IMItation LEarning (SIMILE). The core idea is to introduce sequential information, so that an agent can refer to both the current state and past state-action pairs to make a decision. We formulate our approach into a recurrent model, and instantiate it using LSTM so as to fuse both long-term and short-term information. SIMILE is a generalized IL framework which is easily applied to BL and IRL, two major types of IL algorithms. Experiments are performed on several robot controlling tasks in OpenAI Gym. SIMILE not only achieves performance gain over the baseline approaches, but also enjoys the benefit of faster convergence and better stability of testing performance. These advantages verify a higher learning efficiency of SIMILE, and implies its potential applications in real-world scenarios, i.e., when the agent-environment interaction is more difficult and/or expensive.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 This paper introduces the use of sequential information (state-action pairs) for enhancing imitation learning, and using recurrent networks (LSTM) in that process. The authors motivate this by pointing out that while the state information, if Markovian, should contain all information necessary for decision making, with incomplete learners redundant information in the sequential state-action information leading to the current state can be helpful, citing some concrete examples. After describing a number of variants of this idea, in the context of IRL, BC, etc., the authors conduct a systematic empirical evaluation to assess the effectiveness of the proposal, over the baselines, using a number of RL benchmark problems. The results are favorable and convincingly show that the proposed sequential enhancement can bring significant improvement in terms of attained rewards, convergence speed and stability in many of the tested cases. One suggestion I have is that it would be interesting to investigate into the question of how the addition of sequential information adds value is related to the validity of Markovian assumption in each of the problem being considered. It is a good empirical paper demonstrating the practical use of an idea that is simple but reasonable, and in a way that is substantiated using proper cutting edge framework and baselines.
assistant
 This paper introduces the use of sequential information (state-action pairs) for enhancing imitation learning, and using recurrent networks (LSTM) in that process. The authors motivate this by pointing out that while the state information, if Markovian, should contain all information necessary for decision making, with incomplete learners redundant information in the sequential state-action information leading to the current state can be helpful, citing some concrete examples. After describing a number of variants of this idea, in the context of IRL, BC, etc., the authors conduct a systematic empirical evaluation to assess the effectiveness of the proposal, over the baselines, using a number of RL benchmark problems. The results are favorable and convincingly show that the proposed sequential enhancement can bring significant improvement in terms of attained rewards, convergence speed and stability in many of the tested cases. One suggestion I have is that it would be interesting to investigate into the question of how the addition of sequential information adds value is related to the validity of Markovian assumption in each of the problem being considered. It is a good empirical paper demonstrating the practical use of an idea that is simple but reasonable, and in a way that is substantiated using proper cutting edge framework and baselines.