Structural Causal Bandits: Where to Intervene?
We study the problem of identifying the best action in a sequential decisionmaking setting when the reward distributions of the arms exhibit a non-trivial dependence structure, which is governed by the underlying causal model of the domain where the agent is deployed. In this setting, playing an arm corresponds to intervening on a set of variables and setting them to specific values. In this paper, we show that whenever the underlying causal model is not taken into account during the decision-making process, the standard strategies of simultaneously intervening on all variables or on all the subsets of the variables may, in general, lead to suboptimal policies, regardless of the number of interventions performed by the agent in the environment. We formally acknowledge this phenomenon and investigate structural properties implied by the underlying causal model, which lead to a complete characterization of the relationships between the arms' distributions. We leverage this characterization to build a new algorithm that takes as input a causal structure and finds a minimal, sound, and complete set of qualified arms that an agent should play to maximize its expected reward. We empirically demonstrate that the new strategy learns an optimal policy and leads to orders of magnitude faster convergence rates when compared with its causal-insensitive counterparts.
The authors propose an algorithm that could improve the efficiency of multi-bandits algorithms by selecting a minimal, sound and complete set of possibly optimal arms. In their frameworks arms correspond to interventions on a causal graph, which could possibly have latent confounders. If the structure of the causal graph is known (but not the parameters), the proposed algorithms can exploit this structure to decide which interventions are provably nonoptimal and filter them out.  I think the paper presents some interesting and novel ideas. The clarity could be improved, especially for a reader who is not familiar with the do-calculus literature (e.g. section 2). The experiments are limited, but in my opinion the theory makes the paper interesting in itself.  Minor details: Abstract: I’m a bit confused about how could one empirically demonstrate that an algorithm leads to optimal (...) convergence rates.  Some inconsistency in the notation, e.g.: line 88: PA_i is never defined, although it is clearly pa(X_i),  Lines 117-123: maybe it’s pedantic, but shouldn’t there be some universal quantification on x,y,z, w? For example P(y|do(x), do(z),w) = P(y|do(x),w) for all x \in D(X), z \in D(Z) etc? Line 129: isn’t X = X intersected with an(Y)_{G after do(X)} just X \subset an(Y)_{G after do(X)} ? line 198: pa(T)_G\T … doesn’t pa(.) already exclude itself (in this case T)?  Typos: footnote 4: “that are not latent confounders are not explicitly represented”, line 252 “four strategies”, line 214, 216 “intervening ON”  Appendix B: Instead of Algorithm 4, wouldn’t it be enough to have a much simpler “while (T != oldT) { oldT=T; T=desc(cc(T)); }”?