We describe MÊLÉE, a meta-learning algorithm for learning a good exploration policy in the interactive contextual bandit setting. Here, an algorithm must take actions based on contexts, and learn based only on a reward signal from the action taken, thereby generating an exploration/exploitation trade-off. MÊLÉE addresses this trade-off by learning a good exploration strategy based on offline synthetic tasks, on which it can simulate the contextual bandit setting. Based on these simulations, MÊLÉE uses an imitation learning strategy to learn a good exploration policy that can then be applied to true contextual bandit tasks at test time. We compare MÊLÉE to seven strong baseline contextual bandit algorithms on a set of three hundred real-world datasets, on which it outperforms alternatives in most settings, especially when differences in rewards are large. Finally, we demonstrate the importance of having a rich feature representation for learning how to explore.
This paper investigates a meta-learning approach for the contextual bandit problem. The goal is to learn a generic exploration policy from datasets, and then to apply the exploration policy to contextual bandit tasks. The authors have adapted an algorithm proposed for imitation learning (Ross & Bagnell 2014) to their setting. Some theoretical guarantees straightforwardly extracted from (Ross & Bagnell 2014) and from (Kakade et al 2008) are presented. Experiments are done on 300 supervised datasets.
Major concerns:
1 This paper investigates a problem that does not correspond to the real problem: how to take advantage of a plenty of logs generated by a known stochastic policy (or worst unknown deterministic policy) for the same (or a close) contextual bandit task? 
Most of companies have this problem. I do not know a single use case, in which we have some full information datasets, which are representative of contextual bandit tasks to be performed. If the full information datasets does not correspond to the contextual bandit tasks, it is not possible to learn something useful for the contextual bandit task. 
2 The experimental validation is not convincing.
The experiments are done on datasets, which are mostly binary classification datasets. In this case, the exploration task is easy. May be it is the reason why the exploration parameter \mu or \epsilon = 0 provides the best results for MELEE or \epsilon-greedy?
The baselines are not strong. The only tested contextual bandit algorithm is LinUCB. However a diagonal approximation of the covariance matrix is used when the dimension exceeds 150. In this case LinUCB is not efficient. There are a lot of contextual bandit algorithms that scale with the dimension.
3 The theoretical guarantees are not convincing. 
The result of Theorem 1 is a weak result. A linear regret against the expected reward of the best policy is usually considered as a loosely result. Theorem 2 shows that there is no theoretical gain of the use of the proposed algorithm: the upper bound of the expected number of mistakes obtained when Banditron is used in MELEE is upper than the one of Banditron alone.
Minor concerns:
The algorithms are not well written. POLOPT function has sometimes one parameter, sometimes two and sometimes three parameters. The algorithm 1 is described in section 2, while one of the inputs of the algorithm 1 (feature extractor function) is described in section 3.1. The algorithm 1 seems to return all the N exploration policies. The choice of the returned policy has to be described.
In contextual bandits, the exploration policy is not handcrafted. The contextual bandit algorithms are designed to be optimal or near optimal in worst case: they are generic algorithms.