Combinatorial Cascading Bandits
We propose combinatorial cascading bandits, a class of partial monitoring problems where at each step a learning agent chooses a tuple of ground items subject to constraints and receives a reward if and only if the weights of all chosen items are one. The weights of the items are binary, stochastic, and drawn independently of each other. The agent observes the index of the first chosen item whose weight is zero. This observation model arises in network routing, for instance, where the learning agent may only observe the first link in the routing path which is down, and blocks the path. We propose a UCB-like algorithm for solving our problems, CombCascade; and prove gap-dependent and gap-free upper bounds on its n-step regret. Our proofs build on recent work in stochastic combinatorial semi-bandits but also address two novel challenges of our setting, a non-linear reward function and partial observability. We evaluate CombCascade on two real-world problems and show that it performs well even when our modeling assumptions are violated. We also demonstrate that our setting requires a new learning algorithm.
The authors investigate a class of combinatorial cascading bandit problems. The archetypal problem in this class seems to be that of routing packets in a network where edges are subject to random independent failures. I will restrict to this problem to simplify the description (though the authors also discuss the disjunctive version of this reward function). At each time step the agent chooses a path and receives unit reward only if none of the edges along the path failed. If some edge failed the agent gets a reward of zero and is also given the feedback of which was the first edge that failed. The algorithm they design for this problem called CombCascade, essentially builds UCB estimates for edge failure probability and picks the path with highest expected non-failure probability. They prove upper bounds for the regret of their algorithm one that is gap-dependent and one that is not. They argue that the upper bound matching the previously known tight upper bound for the stochastic semi-bandit problem (which is a more informative feedback setting) perhaps indicates that their algorithm is tight in settings that are "close" to the semi-bandit setting. The paper concludes with some experiments on synthetic and real datasets that show sub-linear growth of regret for their algorithm and they show that it performs better than CombUCB1 which is a stochastic combinatorial semi-bandit algorithm. There are a couple of limitations of the study. One, as the authors also point out, is the rather strong independent failure assumption of the graph edges. [Aside: I appreciate the authors' honest comments pointing out the strongest and weakest parts of their paper, I wish more papers did that, not so much to simplify the referee's task, but to stimulate any reader of the paper.] Secondly, their work points to tantalizing connections to the semi-bandit setting, but this is not clarified well. This is perhaps food for future work. For instance, the weight estimation stage of their algorithm hints at a possible reduction between the two settings. On the other hand, the presence of interesting practical problems in their setting argues for acceptance of the paper despite these issues.