Distributed Multi-Player Bandits -a Game of Thrones Approach
We consider a multi-armed bandit game where N players compete for K arms for T turns. Each player has different expected rewards for the arms, and the instantaneous rewards are independent and identically distributed. Performance is measured using the expected sum of regrets, compared to the optimal assignment of arms to players. We assume that each player only knows her actions and the reward she received each turn. Players cannot observe the actions of other players, and no communication between players is possible. We present a distributed algorithm and prove that it achieves an expected sum of regrets of near-O log 2 T . This is the first algorithm to achieve a poly-logarithmic regret in this fully distributed scenario. All other works have assumed that either all players have the same vector of expected rewards or that communication between players is possible.
This paper studies a distributed multi-armed bandits setting, where every round each of N players must choose one of K arms. If a player picks the same arm as some other player, they receive payoff zero; otherwise, they receive payoff drawn from some distribution (specific to that player and that arm). This can be thought of as learning a distributed allocation or matching from players to arms. The goal is to design a communication-free learning algorithm that maximizes the total overall utility of all the players (or alternatively, minimizes their regret with respect to the best fixed allocation).   The authors design an algorithm which receives total regret of O(log^2 T). Their algorithm can be thought of as a distributed "explore then exploit"-style algorithm (similar to e.g. epsilon-greedy). The algorithm begins with a phase where all players pick random arms to learn their individual distributions, and concludes with a phase where they all repeatedly pick what they have decided is the optimal allocation. The technically interesting part of the algorithm is the intermediate phase, where the arms coordinate amongst themselves (without communication, just by playing the game) to pick a good allocation where no arm is picked by more than one player (this they refer to as the "game of thrones"). They accomplish this by inducing a Markov process between the players of the game which converges to a state where all players are playing according to the optimal allocation. The authors also perform some simulations on synthetic data showing that this algorithm outperforms the case where all players are independently running UCB1.   This is a nice result. That said, it seems like the interesting part of this work is not really the learning component, but rather the coordination component -- in fact, the problem/solution still seems interesting even if all players know their exact expected rewards for each arm ahead of time. From a learning perspective, I would have preferred to see a UCB-style algorithm that explores and exploits at the same time (while somehow simultaneously coming to a consensus on what allocation to play).   Also, it seems important in the analysis that the distribution of rewards for an arm has no point mass at 0, since the way in which players estimate the expected reward for an arm is to just take the average over all non-zero pulls (zero-pulls are assumed to be collisions with other players). Is this necessary to the analysis? One can imagine natural settings where it is not possible to distinguish via the reward alone whether the reward is small because of the distribution or because other players are also playing that arm (either because the distribution has a point mass at 0 or because collisions don't decrease the reward all the way to 0). Does some variant of this algorithm work in this setting? (it is not obvious from the analysis in the paper).  The paper was well-written and easy to understand. I have not checked all the proofs in the supplementary materials, but the proof sketches appear sound.