We propose a method to efficiently learn diverse strategies in reinforcement learning for query reformulation in the tasks of document retrieval and question answering. In the proposed framework an agent consists of multiple specialized sub-agents and a meta-agent that learns to aggregate the answers from sub-agents to produce a final answer. Sub-agents are trained on disjoint partitions of the training data, while the meta-agent is trained on the full training set. Our method makes learning faster, because it is highly parallelizable, and has better generalization performance than strong baselines, such as an ensemble of agents trained on the full data. We show that the improved performance is due to the increased diversity of reformulation strategies.
The authors proposed a variant of ensemble method in reinforcement learning for query reformulation. They train multiple specialized sub-agents on disjoint partitions of the training data, and use a meta-agent, which can see all the training data, to decide the final answer. This can speed up the training thanks to parallelization. They observed that this can improve the diversity of learnt reformulations and the overall performance in some cases. 
Strengths
1. The paper is clear and easy to follow.
2. Multiple evaluation metrics and baseline models are considered
Weaknesses
1. The proposed method is simple and lacks novelty.
2. The performance improvement is marginal and some empirical results are not carefully analyzed. 
Significance
Exploring a diverse set of strategies is beneficial in reinforcement learning. This paper focuses on two aspects of this problem. One is how to learn diverse agents and the other one is how to efficiently learn these agents efficiently, which are important concerns in practice. 
Originality
The model learning approach they proposed is merely a simple variant of ensemble learning. The main difference is they train sub-agents on disjoint partitions of the training data, which seems a trivial modification although this shows to improve the overall model performance.
Technical Quality
Overall, the experiments are well-thought, but the following questions need to be explained:
1.	In the Introduction, the authors claim three contributions they made in this paper. My question is, if the third one is really an important contribution, why didn’t the authors demonstrate it in detail in the main text? Attaching it to the appendix could make the reader confused about its significance.
2.	In Table-1, the authors claim that their proposed architectures can outperform the baseline RL-10-Ensemble with only 1/10 time. The sub-agents are trained on a partition of the training set. My question is, are these sub-agents trained in parallel on different machine? If so, why cannot the RL-10-Ensemble be trained in parallel through some multithread or distributed computation? The implementation of the proposed model and the baseline seems not that fair.
3.	The main architecture is described in section 3.3 and 3.4, including the Sub-agents and the Aggregator. However, in Appendix C.1, the authors claim that the gains the proposed method comes mostly from the pool of diverse reformulators, and not from the simple use of a re-ranking function (Aggregator). This is confusing because if it is true, the proposed method is really reduced to an ensemble of the baseline model.
4.	In Table-2, some of the results are worse than the baseline methods like Re-Ranker. Although the authors claim the re-ranking is a post-processing, Re-Ranker performs significantly better than the proposed model. If the authors want to better demonstrate the advantages of the proposed model, a comparison between the proposed model with re-ranking and the Re-Ranker is required.
5.	In Table 10, why the proposed method fails to produce the right answer whereas the other methods perform well?