Bandits with Unobserved Confounders: A Causal Approach
The Multi-Armed Bandit problem constitutes an archetypal setting for sequential decision-making, permeating multiple domains including engineering, business, and medicine. One of the hallmarks of a bandit setting is the agent's capacity to explore its environment through active intervention, which contrasts with the ability to collect passive data by estimating associational relationships between actions and payouts. The existence of unobserved confounders, namely unmeasured variables affecting both the action and the outcome variables, implies that these two data-collection modes will in general not coincide. In this paper, we show that formalizing this distinction has conceptual and algorithmic implications to the bandit setting. The current generation of bandit algorithms implicitly try to maximize rewards based on estimation of the experimental distribution, which we show is not always the best strategy to pursue. Indeed, to achieve low regret in certain realistic classes of bandit problems (namely, in the face of unobserved confounders), both experimental and observational quantities are required by the rational agent. After this realization, we propose an optimization metric (employing both experimental and observational distributions) that bandit agents should pursue, and illustrate its benefits over traditional algorithms.
The paper "Bandits with unobs. confounders: a causal approach" addresses the problem of bandit learning. It is assumed that in the observational setting, the player's decision is influenced by some unobserved context. If we randomize the player's decision, however, this intention is lost. The key idea is now that, using the available data from both scenarios, one can infer whether one should overrule the player's intention. Ultimately, this leads to the following strategy: observe the player's intention and then decide whether he should act accordingly or pull the other arm.  I like the problem the paper addresses and the way it approaches it. Although I would have liked to do so, however, I cannot strongly recommend the acceptance of this paper due to1) it contains almost no material beyond an idea 2) limitations to bivariate variables? 3) method is not properly compared to contextual bandits, experiments available only on two artificial toy data sets I definitely encourage the authors to continue this line of research and provide a more thorough discussion of this problem.- the key idea seems to be (7), which computes a counterfactual statement from observational and interventional data. unfortunately, this is already known (e.g. "What Counterfactuals Can Be Tested" Shpitser, Pearl, p.353).- also, i am not sure how the idea generalizes to variables with more than 2 states (i.e., #arms and/or pay-off). it is known that in general, counterfactuals cannot be inferred from obs. and int. data. This seems to be a major limitation that is not discussed properly.- the assumption that the intention of the player can be measured before he pulls an arm is quite strong. but if it can, this problem is "just" a contextual bandit (e.g. use TS^Z). It remained unclear to me, why in the simulations, the proposed method got access to the observational data, while the TS^Z approach did not (if it did, it should be at least as good!?). I think, one could rewrite the whole motivation of the paper: consider a situation, in which one first observes a _different_ player (obs. data) playing a bandit and then oneself sits down in order to gather the interventional data. this requires the assumption that all players develop the same intention. but then it seems to boil down to a contextual bandit with a context that is partially hidden for some data points. Is there any literature on this problem? - the first concrete idea (as far as I can tell) is in the last paragraph on page 6 (for a conference paper, the introduction is too long)- there is no code available- there are only toy examples- in many places the paper is badly written/unclear. i only give few examples, there are more * bottom page 2, there are two machines (sometimes two arms), but D indicates whether "the" machine is blinking * Y = {0,1} doesn't make sense if Y is a rand.var. * the authors confuse E and P many times. E(Y) = P(Y=1) if Y takes values in {0,1} and change notation. * alg. 1 contains many unclear statements, e.g. beta and f are not defined properly, what is x' etc. * the writing f(x,a) for all a = x, is not very sensible, bottom p. 6 * the abstract is a bit unclear "passive data through random sampling"- maybe focus more on the example?- the colors in fig. 1 are indistinguishable minor - greendy casino - use \eqref - shouldn't (8) be <=> ? - is great than - citations are very inhomogeneous - random experiments do not control for but eliminate the influence of hidden confounders. - what are the grey lines in Fig. 2? - I would not regard Def. 3.3. as a definition