Stochastic Online Greedy Learning with Semi-bandit Feedbacks
The greedy algorithm is extensively studied in the field of combinatorial optimization for decades. In this paper, we address the online learning problem when the input to the greedy algorithm is stochastic with unknown parameters that have to be learned over time. We first propose the greedy regret and -quasi greedy regret as learning metrics comparing with the performance of offline greedy algorithm. We then propose two online greedy learning algorithms with semi-bandit feedbacks, which use multi-armed bandit and pure exploration bandit policies at each level of greedy learning, one for each of the regret metrics respectively. Both algorithms achieve O(log T ) problem-dependent regret bound (T being the time horizon) for a general class of combinatorial structures and reward functions that allow greedy solutions. We further show that the bound is tight in T and other problem instance parameters.
Strengths:1) Technical and well executed. Upper bounds and a matching lower bound.2) Pleasure to read.3) A general class of problems which is of interest in large-scale learning.Weaknesses:1) Positioning with respect to the prior work is insufficient and needs to be improved.2) Not particular novel and surprising, given the vast amount of work on related subjects in the past few years.Major comments:This paper cites most papers from the past few years on related subjects. However, it falls short in discussing details in which this work differs from the past work, such as the rates of regret bounds and the computational complexity of the proposed methods. This needs to be improved. This is my point of view. Matroids and polymatroids are generally regarded as the largest classes of problems where the greedy algorithm is guaranteed to be optimal. [23] and:   Kveton, Branislav, Wen, Zheng, Ashkan, Azin, and Valko, Michal. Learning to act greedily: Polymatroid semi-bandits. CoRR, abs/1405.7752, 2014.which I further refer to as [33], proposed optimal learning algorithms for these problems. The main difference in this work is that the optimized function is not modular and that the feasible set is more general. In this setting, the greedy algorithm is not guaranteed to be optimal, or even to be a good approximation. This should be highlighted. Nevertheless, this work allows for solving new and interesting learning problems, such as learning to maximize a submodular function subject to a cardinality constraint in the semi-bandit setting.In the notation of this paper, [23] and [33] prove O(n (1 / \Delta) \log n) upper bounds on the regret of their learning algorithms. Let's compare this to Theorem 3.1. Clearly W = O(n) and \Delta_{max} = O(m). Therefore, the upper bound in Theorem 3.1 is O(m^2 n (1 / \Delta^2) log n), a factor of m^2 (1 / \Delta) larger than the result of [23] on more specific problems. Perhaps this is not surprising, given how general the proposed framework is, but it should be mentioned that OG is less statistically efficient than OMM [23] on many practical problems, such as learning a minimum spanning tree. The authors should also discuss the space complexity of OG. As far as I can tell, it is linear in |F|, which is n \choose m when the feasible set is a uniform matroid of rank m. Therefore, OG is not space efficient on many practical problems. When the gaps are small, even lazy initialization (line 269) could lead to exploring all n \choose m arms. In comparison, the space complexity of OMM [23] on a uniform matroid of rank m is O(n).Finally, note that [24] prove a O(m n (1 / \Delta') \log n) upper bound on the regret of CombUCB1 in stochastic combinatorial semi-bandits, where \Delta' is arguably larger than \Delta in this paper. Therefore, the upper bound of [24] is a factor of m (1 / \Delta) smaller than that in Theorem 3.1. This upper bound holds for any linear objective on any feasible set, and even for approximation oracles. Therefore, CombUCB1 is more statistically efficient than OG in this setting. The above also indicates that the extra factors in Theorem 3.1 may be due to the non-linearity of the optimized objective. This is worth mentioning.Based on the above, the novelty of this work is mostly in learning suboptimal greedy policies for non-linear objectives. The proposed learning algorithms are not guaranteed to be space efficient. Therefore, the impact of this work on practice may be limited. To make it more significant, I suggest the authors include a good motivating example, which showcases the importance of learning good suboptimal greedy policies, which can be learned efficiently by either OG or LUCB. Experimental results on this problem would be nice and convince me that the class of problems in this paper is important and worth studying. The current motivating example is bad in the sense that this paper does not solve this particular problem.My last major comment is that the lower bound in Section 5 is not very clear. In particular, I am not sure how the authors conclude that:   \Delta + 0.5 = \Delta_{max} + o(1).It can be also argued that the case where \epsilon < \Delta is not very interesting.Minor comments:Line 128: It may be less ambiguous to write ordered sets as vectors and then abuse the set notation when needed. For instance, \sigma = (S_0, S_1, \dots, S_k) instead of \sigma = \{S_0, S_1, \dots, S_k\}.Line 191: "online algorithm that are" should read as "online algorithms that are".Line 192: "objective is to minimum" should read as "objective is to minimize".