summaries: The paper proposes to learn interventional distributions using a family of generative models known as sum-product networks (SPN). SPNs are models based on gate functions that allow for tractable inference. Specifically, the authors propose interventional SPNs (iSPN) which are conditional SPNs (they model conditional distributions) that, licensed by a causal graph and samples from the interventional distribution, are shown to be expressive and causally adequate. The paper proposes a new causal inference framework based on so-called ‘interventional Sum-Product Networks’ to target the problem of a lack of tractability in causal inference. It is based on standard SPNs and employs deep learning techniques to capture complex nonparametric functions describing multivariate conditional probability distributions. It is evaluated on several small synthetic data sets.  This paper proposes a modification of sum product networks termed iSPN to estimate interventional distributions from data. The proposed model performs a manipulation to the structure of the SPN in order to provide estimates from the mutilated distribution. The authors motivate the use of this model in terms of computational complexity and the ability to represent estimates of arbitrary causal quantities on the graph. Empirical results show the proposed model having impressive running time and comparable error results to modern software packages for causal effect estimation.  This submission proposes interventional sum-product networks, a tractable method for estimating causal effects from interventional data. The authors prove that iSPNs are universal function approximators. They show that iSPNs produce accurate estimates of interventional queries from interventional data on collection of synthetic data generating processes. This paper discusses a variant of sum-product networks (SPNs) called "interventional sum-product networks" which is a variant of SPNs that take as input a causal graph and use this to model interventional distributions. The claim is that this is the first paper that applies tractable probabilistic models to causality. Theoretical aspects are discussed and the method is evaluated and compared to existing methods. 
abstractText: While probabilistic models are an important tool for studying causality, doing so suffers from the intractability of inference. As a step towards tractable causal models, we consider the problem of learning interventional distributions using sumproduct networks (SPNs) that are over-parameterized by gate functions, e.g., neural networks. Providing an arbitrarily intervened causal graph as input, effectively subsuming Pearl’s do-operator, the gate function predicts the parameters of the SPN. The resulting interventional SPNs are motivated and illustrated by a structural causal model themed around personal health. Our empirical evaluation against competing methods from both generative and causal modelling demonstrates that interventional SPNs indeed are both expressive and causally adequate.
authors.name: Matej Zečević
authors.name: Devendra Singh Dhami
authors.name: Athresh Karanam
authors.name: Sriraam Natarajan
authors.name: Kristian Kersting
references.authors: Jayadev Acharya Arnab Bhattacharyya Constantinos Daskalakis Saravanan Kandasamy
references.title: Learning and testing causal models with interventions
references.venue: NeurIPS,
references.authors: Elias Bareinboim Judea Pearl
references.title: Causal inference and the data-fusion problem
references.authors: Sander Beckers Joseph Y Halpern
references.title: Abstracting causal models
references.venue: In AAAI,
references.authors: Ioana Bica Ahmed Alaa Mihaela Van Der Schaar
references.title: Time series deconfounder: Estimating treatment effects over time in the presence of hidden confounders
references.venue: In ICML,
references.authors: Christopher M Bishop
references.title: Mixture density networks
references.authors: Philippe Brouillard Sébastien Lachapelle Alexandre Lacoste Simon Lacoste-Julien Alexandre Drouin
references.title: Differentiable causal discovery from interventional data
references.venue: NeurIPS,
references.authors: John F Carriger Mace G Barron
references.title: A bayesian network approach to refining ecological risk assessments: Mercury and the florida panther (puma concolor coryi)
references.venue: Ecological Modelling,
references.authors: Daniel C Castro Jeremy Tan Bernhard Kainz Ender Konukoglu Ben Glocker
references.title: Morpho-mnist: quantitative assessment and diagnostics for representation learning
references.venue: Journal of Machine Learning Research,
references.authors: Huigang Chen Totte Harinen Jeong-Yoon Lee Mike Yung Zhenyu Zhao
references.title: Causalml: Python package for causal machine learning
references.venue: arXiv preprint arXiv:2002.11631,
references.authors: Arthur Choi Adnan Darwiche
references.title: On the relative expressiveness of bayesian and neural networks
references.venue: In International Conference on Probabilistic Graphical Models,
references.authors: Gregory F Cooper
references.title: The computational complexity of probabilistic inference using bayesian belief networks
references.venue: Artificial intelligence,
references.authors: Adnan Darwiche
references.title: A differential approach to inference in bayesian networks
references.venue: Journal of the ACM (JACM),
references.authors: Ishita Dasgupta Jane Wang Silvia Chiappa Jovana Mitrovic Pedro Ortega David Raposo Edward Hughes Peter Battaglia Matthew Botvinick Zeb Kurth-Nelson
references.title: Causal reasoning from meta-reinforcement learning
references.authors: Norman E Fenton Martin Neil Magda Osman Scott McLachlan
references.title: Covid-19 infection and death rates: the need to incorporate causal explanations for the data and avoid bias in testing
references.venue: Journal of Risk Research,
references.authors: Navid Feroze
references.title: Forecasting the patterns of covid-19 and causal impacts of lockdown in top five affected countries using bayesian structural time series models
references.venue: Chaos, Solitons & Fractals,
references.authors: Mathieu Germain Karol Gregor Iain Murray Hugo Larochelle
references.title: Made: Masked autoencoder for distribution estimation
references.venue: In International Conference on Machine Learning,
references.authors: Clive WJ Granger
references.title: Investigating causal relations by econometric models and cross-spectral methods
references.venue: Econometrica: journal of the Econometric Society,
references.authors: York Hagmayer Steven A Sloman David A Lagnado Michael R Waldmann
references.title: Causal reasoning through intervention
references.venue: Causal learning: Psychology, philosophy, and computation,
references.authors: Joseph F Hair Jr. Marko Sarstedt
references.title: Data, measurement, and causal inferences in machine learning: opportunities and challenges for marketing
references.venue: Journal of Marketing Theory and Practice,
references.authors: David Heckerman Dan Geiger David M Chickering
references.title: Learning bayesian networks: The combination of knowledge and statistical data
references.venue: Machine learning,
references.authors: William Hoiles Mihaela Schaar
references.title: Bounded off-policy evaluation with missing data for course recommendation and curriculum design
references.venue: In ICML,
references.authors: Agastya Kalra Abdullah Rashwan Wei-Shou Hsu Pascal Poupart Prashant Doshi Georgios Trimponias
references.title: Online structure learning for feed-forward and recurrent sum-product networks. NeurIPS, 2018
references.authors: Nan Rosemary Ke Olexa Bilaniuk Anirudh Goyal Stefan Bauer Hugo Larochelle Bernhard Schölkopf Michael C Mozer Chris Pal Yoshua Bengio
references.title: Learning neural causal models from unknown interventions
references.authors: Doga Kisa Guy Van den Broeck Arthur Choi Adnan Darwiche
references.title: Probabilistic sentential decision diagrams
references.venue: In KR,
references.authors: Daniel Koch Robert S Eisinger Alexander Gebharter
references.title: A causal bayesian network model of disease progression mechanisms in chronic myeloid leukemia
references.venue: Journal of theoretical biology,
references.authors: Kevin B Korb Ann E Nicholson
references.title: Bayesian artificial intelligence
references.venue: CRC press,
references.authors: Steffen L Lauritzen David J Spiegelhalter
references.title: Local computations with probabilities on graphical structures and their application to expert systems
references.venue: Journal of the Royal Statistical Society: Series B (Methodological),
references.authors: Amélie Levray Vaishak Belle
references.title: Learning credal sum-product networks
references.venue: AKBC,
references.authors: Stephen L Morgan Christopher Winship
references.title: Counterfactuals and causal inference
references.authors: Richard E. Neapolitan
references.title: Learning bayesian networks
references.venue: Pearson Prentice Hall Upper Saddle River, NJ,
references.authors: Mike Oaksford Nick Chater
references.title: Causal models and conditional reasoning
references.venue: Oxford library of psychology. The Oxford handbook of causal reasoning,
references.authors: Michael Oberst David Sontag
references.title: Counterfactual off-policy evaluation with gumbel-max structural causal models
references.venue: In ICML,
references.authors: Ioannis Papantonis Vaishak Belle
references.title: Interventions and counterfactuals in tractable probabilistic models: Limitations of contemporary transformations
references.venue: arXiv preprint arXiv:2001.10905,
references.authors: Iago París Raquel Sánchez-Cauce Francisco Javier Díez
references.title: Sum-product networks: A survey
references.venue: arXiv preprint arXiv:2004.01167,
references.authors: Judea Pearl
references.title: From bayesian networks to causal networks. In Mathematical models for handling partial knowledge in artificial intelligence
references.authors: Judea Pearl
references.title: The seven tools of causal inference, with reflections on machine learning
references.venue: Communications of the ACM,
references.authors: Judea Pearl Madelyn Glymour Nicholas P Jewell
references.title: Causal inference in statistics: A primer
references.authors: Robert Peharz Antonio Vergari Karl Stelzner Alejandro Molina Xiaoting Shao Martin Trapp Kristian Kersting Zoubin Ghahramani
references.title: Random sum-product networks: A simple and effective approach to probabilistic deep learning
references.venue: In UAI,
references.authors: Jonas Peters Dominik Janzing Bernhard Schölkopf
references.title: Elements of causal inference
references.authors: Hoifung Poon Pedro Domingos
references.title: Sum-product networks: A new deep architecture
references.venue: IEEE International Conference on Computer Vision Workshops (ICCV Workshops),
references.authors: Dan Roth
references.title: On the hardness of approximate reasoning
references.venue: Artificial Intelligence,
references.authors: Bernhard Schölkopf
references.title: Causality for machine learning
references.venue: arXiv preprint arXiv:1911.10500,
references.authors: Karthikeyan Shanmugam Murat Kocaoglu Alexandros G Dimakis Sriram Vishwanath
references.title: Learning causal graphs with small interventions
references.venue: NeurIPS,
references.authors: Xiaoting Shao Alejandro Molina Antonio Vergari Karl Stelzner Robert Peharz Thomas Liebig Kristian Kersting
references.title: Conditional sum-product networks: Imposing structure on deep probabilistic architectures
references.authors: Amit Sharma Emre Kiciman
references.title: Dowhy: An end-to-end library for causal inference
references.venue: arXiv preprint arXiv:2011.04216,
references.authors: Edward H Simpson
references.title: The interpretation of interaction in contingency tables
references.venue: Journal of the Royal Statistical Society: Series B (Methodological),
references.authors: G Van den Broeck N Di Mauro A Vergari
references.title: Tractable probabilistic models: Representations, algorithms, learning, and applications
references.venue: Tutorial at UAI,
references.authors: Kevin Xia Kai-Zhan Lee Yoshua Bengio Elias Bareinboim
references.title: The causal-neural connection: Expressiveness
references.venue: learnability, and inference
references.authors: Kun Zhang Bernhard Schölkopf Peter Spirtes Clark Glymour
references.title: Learning causality and causalityrelated learning: some recent progress
references.venue: National science review,
references.authors: Han Zhao Mazen Melibari Pascal Poupart
references.title: On the relationship between sum-product networks and bayesian networks
references.venue: In ICML,
sections.heading: 1 Introduction
sections.text: Identifying causal relationships between variables in observational data is one of the fundamental and well-studied problem in machine learning. There have been several great strides in causality [Granger, 1969, Pearl, 2009, Bareinboim and Pearl, 2016] over the years specifically characterized by efforts that focused on reasoning about interventions [Hagmayer et al., 2007, Dasgupta et al., 2019] and counterfactuals [Morgan and Winship, 2015, Oberst and Sontag, 2019].
  The notion of causality has long been explored in the realm of probabilistic models [Oaksford and Chater, 2017, Beckers and Halpern, 2019] with a special focus on graphical models, called causal Bayesian networks (CBNs) [Heckerman et al., 1995, Neapolitan, 2004, Pearl, 1995, Acharya et al., 2018]. CBNs have widely been applied to infer causal relationships in high-impact diverse applications such as disease progression [Koch et al., 2017], ecological risk assessment [Carriger and Barron, 2020] and more recently Covid-19 [Fenton et al., 2020, Feroze, 2020] to name a few. Although successful, classical CBN models are difficult to scale and also suffer from the problem of intractable inference. Recently, tractable probabilistic models such as probabilistic sentential decision diagrams [Kisa et al., 2014] and sum-product networks [Poon and Domingos, 2011] have emerged, which guarantee that conditional marginals can be computed in time linear in the size of the model. While weaving in the notion of interpretability, the computational view on probabilistic models allows one to exploit ideas from deep learning and can thus be very useful in modelling complex problems.
  Recently, there has been an effort to take advantage of this tractability to reason for causality. Zhao et al. [2015] showed how to compile back and forth between sum-product networks (SPNs) and Bayesian networks (BNs). Although this opened up a whole range of possibilities for tractable
  35th Conference on Neural Information Processing Systems (NeurIPS 2021), virtual.
  causal models, Papantonis and Belle [2020] argued that such conversion leads to degenerated BNs thereby rendering it ineffective for causal reasoning. For the considered compilation of SPNs to BNs, this is indeed the case since a bipartite graph between the hidden and observed variables loses the relationships between the actual variables. Thus, either a new compilation method for transforming between tractable and causal model or alternatively a method to condition the probabilistic models directly on the do-operator to obtain interventional distributions, P (y| do(x)), is being required. Here, we consider the latter strategy and extend the idea of conditionally parameterizing SPNs [Shao et al., 2019] by conditioning on the do-operator while predicting the complete set of observed variables therefore capturing the effect of intervention(s). The resulting interventional sum-product networks (iSPNs) take advantage of both the expressivity, due to the neural network, and the tractability, due to the SPN in order to capture the interventional distributions faithfully. This shows that the dream of tractable causal models is not insurmountable, since iSPNs are causal. Pearl [2019] defined a three-level causal hierarchy that separates association (purely statistical level 1) from intervention (level 2) and counterfactuals (level 3), and argued that the latter two levels involve causal inference. iSPNs are a modelling scheme for arbitrary interventional distributions. They thus belong to level 2 and, in turn, are causal. So, while SPNs are not universal function approximators and the use of gate functions turns them into universal approximators, we go one step ahead and make the first effort towards introducing causality to SPNs without the need for compilation to Bayesian networks, as the functional approximator subsumes the do-operator. Fig. 1 shows an example of the effectiveness of iSPNs to capture interventional distributions on the ASIA data set. Our extensive experiments against strong baselines demonstrate that our method is able to capture ideal interventional distributions.
  To summarize, we make the following contributions:
  1. We introduce iSPNs, the first method that applies the idea of tractable probabilistic models to causality without the need for compilation and that accordingly generate interventional distributions w.r.t. the provided causal structure.
  2. We formulate the inductive bias necessary for turning conditional SPN into iSPN while taking advantage of the neural network modelling capacities within the gating nodes for for modelling interventional distributions while capturing all influences within the given intervention (i.e., consequences propagated through the structural hierarchy).
  3. We show that, by construction, iSPNs can identify any interventional distribution permitted by the underlying structural causal model due to the inducted bias on the interface modalities in junction with the universal function approximation of the underlying gated SPN.
  We proceed as follows. We start by reviewing the basic concepts required and related work, namely the tractable model class of sum-product networks and key concepts from causality. Then we motivate using a newly curated causal model themed around personal health. Subsequently, we introduce iSPNs formally and prove that by construction they are capable of approximating any do-query (given corresponding data). Before concluding, we present our experimental evaluation in which we
  challenge iSPN in its density estimation and causal inference capabilities against various baselines. We make our code publically available at: https://github.com/zecevic-matej/iSPN.
sections.heading: 2 Background and Related Work
sections.text: Let us briefly review the background on tractable probabilistic models and causal models used in subsequent sections for developing our new model class based on CSPNs that allow for identifying causal quantities i.e., interventional distributions.
  Notation. We denote indices by lower-case letters, functions by the general form g(·), scalars or random variables interchangeably by upper-case letters, vectors, matrices and tensors with different boldface font v,V,V respectively, and probabilities of a set of random variables X as p(X).
  Sum-Product Networks (SPNs). Introduced by Poon and Domingos [2011], generalizing the notion of network polynomials based on indicator variables λX=x(x) ∈ [0, 1] for (finite-state) RVs X from [Darwiche, 2003], Sum-Product Networks (SPNs) represent a special type of probabilistic model that allows for a variety of exact and efficient inference routines. Generally, SPNs are considered as directed acyclic graphs (DAG) consisting of product, sum and leaf (or distribution) nodes whose structure and parameterization can be efficiently learned from data to allow for efficient modelling of joint probability distributions p(X). Formally a SPN S = (G,w) consists of non-negative parameters w and a DAG G = (V,E) with indicator variable λ leaf nodes and exclusively internal sum and product nodes given by,
  S(λ) = ∑
  C∈ch(S)
  wS,CC(λ) P(λ) = ∏
  C∈ch(S)
  C(λ), (1)
  where the SPN output S is computed at the root node (S(λ) = S(x)) and the probability density for x is p(x) = S(x)∑
  x′∈X S(x′) . They are members of the family of probabilistic circuits [Van den
  Broeck et al., 2019]. A special class, to be precise, that satisfies properties known as completeness and decomposability. Let N denote a node in SPN S, then
  sc(N) =
  { {X} if N is IV (λX=x)⋃
  C∈ch(N) sc(C) else (2)
  is called the scope of N and
  ∀S ∈ S : (∀C1,C2 ∈ ch(S) : sc(C1) = sc(C2)) (3) ∀P ∈ S : (∀C1,C2 ∈ ch(S) : C1 ̸= C2 =⇒ sc(C1) ∩ sc(C2) = ∅) (4)
  are the completeness and decomposability properties respectively. Since their introduction, SPNs have been heavily studied such as by [Trapp et al., 2019] that present a way to learn SPNs in a Bayesian realm whereas [Kalra et al., 2018] learn SPNs in an online setting. Several different types of SPNs have also been studied such as Random SPN [Peharz et al., 2020], Credal SPNs [Levray and Belle, 2020] and Sum-Product-Quotient Networks [Sharir and Shashua, 2018]) to name a few. For more details readers are referred to the survey of París, Sánchez-Cauce, and Díez [2020]. On another note, Gated or Conditional SPNs (CSPNs) are deep tractable models for estimating multivariate, conditional probability distributions p(Y|X) over mixed variables Y [Shao et al., 2019]. They introduce functional gate nodes gi(X) that act as a functional parameterization of the SPN’s information flow and leaf distributions given the provided evidence X.
  Causal Models. A Structural Causal Model (SCM) as defined by Peters et al. [2017] is specified as C := (S, PN) where PN is a product distribution over noise variables and S is defined to be a set of d structural equations
  Xi := fi(pa(Xi), Ni), where i = 1, . . . , d (5)
  with pa(Xi) representing the parents of Xi in graph G(C). An intervention on a SCM C as defined in (5) occurs when (multiple) structural equations are being replaced through new non-parametric functions f̂(p̂a(Xi), N̂i) thus effectively creating an alternate SCM Ĉ. Interventions are referred to as imperfect if p̂a(Xi) = pa(Xi) and as atomic if f̂ = a for a ∈ R. An important property
  of interventions often referred to as "modularity" or "autonomy"1 states that interventions are fundamentally of local nature, formally
  pC(Xi | pa(Xi)) = pĈ(Xi | pa(Xi)) , (6)
  where the intervention of Ĉ occured on variable Xk opposed to Xi. This suggests that mechanisms remain invariant to changes in other mechanisms which implies that only information about the effective changes induced by the intervention need to be compensated for. An important consequence of autonomy is the truncated factorization
  p(V ) = ∏
  i/∈S p(Xi | pa(Xi)) (7)
  derived by Pearl [2009], which suggests that an intervention S introduces an independence of an intervened node Xi to its causal parents. Another important assumption in causality is that causal mechanisms do not change through intervention suggesting a notion of invariance to the cause-effect relations of variables which further implies an invariance to the origin of the mechanism i.e., whether it occurs naturally or through means of intervention [Pearl et al., 2016].
  A SCM C is capable of emitting various mathematical objects such as graph structure, statistical and causal quantities placing it at the heart of causal inference, rendering it applicable to machine learning applications in marketing [Hair Jr and Sarstedt, 2021]), healthcare [Bica et al., 2020]) and education [Hoiles and Schaar, 2016]. A SCM induces a causal graphG, an observational/associational distribution pC, can be intervened upon using the do-operator and thus generate interventional distributions pC;do(...) and given some observations v can also be queried for interventions within a system with fixed noise terms amounting to counterfactual distributions pC|V=v;do(...). To query for samples of a given SCM, the structural equations are being simulated sequentially following the underlying causal structure starting from independent, exogenous variables and then moving along the causal hierarchy of endogenous variables (i.e., following the causal descendants).
  The work closest to our work is by Brouillard et al. [2020] although it solves the different problem of causal discovery. Causality for machine learning has recently gained a lot of traction [Schölkopf, 2019] with the study of both interventions [Shanmugam et al., 2015] and counterfactuals [Kusner et al., 2017] gaining speed. For more details the readers are referred to [Zhang et al., 2018].
sections.heading: 3 Interventional SPNs
sections.text: Now we are ready to develop interventional SPNs (iSPNs). To this end, we re-introduce the importance of adaptability of models to interventional queries and present a newly curated synthetic data set to both motivate and validate the formalism of iSPNs that, through over-parametric extension of SPNs, allows them to adhere to causal quantities.
sections.heading: 3.1 Adaptation to Causal Change
sections.text: Peters et al. [2017] motivated the necessity of causality via the adequate generalizability of predictive models. Specifically, consider a simple regression problem f(a) = b with data vectors a,b ∈ Rk that are strongly positively correlated in some given region, e.g. (1 < a <∞, 1 < b <∞). Now a query is posed outside the data support, e.g. (0, f(0)). As argued by Peters et al., the underlying data generating processes can be an ambiguous causal process i.e., the data at hand can be explained by two different causal structures being either A→ B or a common confounder with A← C → B. Assuming the wrong causal structure or ignoring it altogether could be fatal, therefore, any form of generalization out of data support requires assumptions to be made about the underlying causal structure. We adopt this point of view and further argue that ignoring causal change(s) in a system, i.e., the change of structural equation(s) underlying the system, can lead to a significant performance decrease and safety hazards2. Therefore, it is important to account for distributional changes present in the data due to experimental (thus interventional) settings.
  1See Section 6.6 in [Peters et al., 2017]. 2This extended notion of performance degeneration through ignorance to the underlying causality is priori-
  tized in this paper.
  Consequently, we consider the learning problem, where the given data samples have been generated by different interventions do(Uj = uj) in a common SCM C while the induced mutilated causal graphs G(C, do(Uj = uj)) are assumed to be known, such that the trained model is capable of at least inferring all involved causal distributions p(V (C) | do(Uj = uj)) with V being the variables.
sections.heading: 3.2 Data Generating Process
sections.text: To both validate and demonstrate the expressivity of interventional SPNs in modelling arbitrary interventional distributions, we curate a new causal data set based on the SCM C presented in Fig. 2(a), which we subsequently refer to as Causal Health data set. The SCM encompasses four different structural equations of the form Vi = f(pa(Vi), Ni), where pa(Vi) are the parents of variable Vi and Ni are the respective noise terms that form a factor distribution PN1:N i.e., the Ni are jointly independent. Now, the SCM C describes the causal relations of an individual’s health and mobility attributes with respect to their age and nutrition.
  Note that the Causal Health data set does not impose assumptions over the type of random variables or functional domains of the structural equations3, which additionally constraints a learned model to adapt flexibly. While we generally do not restrict our method to any particular type of intervention, the following mainly considers perfect interventions as introduced in Sec. 2.
  Perfect interventions fully remove the causal mechanism of the parents of a given node, which is consistent with the idea behind randomized controlled trials (RCTs) where the given intervention randomizes the given specific node, often referred to as gold standard in causality related literature. We consider the special case of uniform randomization, i.e., uniform across the domain of the given variable4. An intervention performed on one node immediately changes the population and, thus, has a major effect on the generating processes of subsequent causal mechanisms in the respective causal sequence of events.
  To provide the reader with a concrete example of interventions within the causal health data set, consider the following: In concern of a virus infection the individuals of the Causal Health study should be vaccinated. The vaccine is expected to have side-effect(s), however, it has been poorly designed and has reached the population with the capability of completely changing the individual’s
  3Assumptions on the func. form of structural eq. are crucial for identification (Tab. 7.1 [Peters et al., 2017]) 4Note that for binary variables this amounts to Bernoulli B( 1
  2 ).
  health state. The observed changes do not show any form of pattern and are therefore assumed to be random. A young fit person could thus become sick, while an old person might feel better health wise. This sudden change will have an effect on the individual’s mobility and also be independent of their age and nutrition. Such a scenario is mathematically being captured through do(H = U(H)) where U(·) is the uniform distribution over a given domain.
sections.heading: 3.3 Introducing Interventional SPNs
sections.text: After motivating both the importance and the occurrences of interventions within relevant systems, we now start introducing interventional SPNs (iSPNs).
  Definition of iSPN. As motivated in Sec. 1, the usage of the compilation method from [Zhao et al., 2015] for causal inference within SPN is arguably of degenerate5 nature given the properties of the compilation method [Papantonis and Belle, 2020]. While the results of Papantonis and Belle [2020] are arguably negative, there exists yet no proof of non-existence of such a compilation method and as the authors point out in their argument for future lines of research in this direction, a model class extension poses a viable candidate for overcoming the problems of using SPN for causal inference.
  While agreeing on the latter aspect, we do not go the “compilation road” but extend the idea of conditional parameterization for SPN [Shao et al., 2019] by conditioning on a modified form of the do-operator introduced by Pearl [2009] while predicting the complete set of observed variables.
  Mathematically, we estimate p(Vi | do(Uj = uj)) by learning a non-parametric function approximator f(G;θ) (e.g. neural network), which takes as input the (mutilated) causal graph G ∈ {0, 1}N×N encoded as an adjacency matrix, to predict the parameters ψ of a SPN g(D;ψ) that estimates the density of the given data matrix {Vk}Kk = D ∈ RK×N . With this, iSPNs are defined as follows: Definition 1 (Interventional Sum-Product Network). An interventional sum-product network (iSPN) is the joint model m(G,D) = g(D;ψ = f(G;θ)), where g(·) is a SPN, f(·) a non-parametric function approximator and ψ = f(G) are shared parameters.
  They are called interventional because we consider it to be a causal model given its capability of answering queries from the second level of the causal hierarchy [Pearl, 2019], namely, that of interventions6. The shared parameters ψ allow for information flow during learning between the conditions and the estimated densities. Setting the conditions such that they contain information about the interventions, in the form of the mutilated graphs G, effectively renders f to subsume a sort of do-calculus in the spirit of truncated factorization shown in Eq.(7) i.e., the gate model acts as an estimand selector. Generally, we note that our formulation allows for different function and density estimators f, g. We choose f to be a neural network for two reasons (1) their empirically established capability to act as causal sub-modules (e.g. Ke et al. [2019] use a cohort of neural nets to mimic a set of structural equations, thus, a SCM) and (2) their model capacity being a universal function approximator, while we choose g to be a SPN for its tractability properties for inference.
  We have argued the importance of adaptability to interventional changes within the causal system and intend now to prove that iSPN are capable of approximating these different causal quantities.
  Proposition 1 (Expressivity). Assuming autonomy and invariance, an iSPN m(G,D) is able to identify any interventional distribution pG(Vi = vi | do(Uj = uj)), permitted by a SCM C through interventions, with knowledge of the mutilated causal graph Ĝ and data D generated from the intervened SCMs by modelling the conditional distribution pĜ(Vi = vi | Uj = uj).
  Proof. It follows directly from the definition of the do-calculus [Pearl, 2009] that pG(Vi = vi | do(Uj = uj)) = pĜ(Vi = vi | Uj = uj) where Ĝ is the mutilated causal graph according to the intervention do(Uj = uj) i.e., observations in the intervened system are akin to observations made when intervening on the system. Given the mutilated causal graph Ĝ (as adjacency matrix), the only remaining aspect to show is that the density estimating SPN can approximate a joint probability p(X) using D. This naturally follows from [Poon and Domingos, 2011].
  5A bipartite graph in which the actual variables of interest are not connected is called degenerate. 6The first level of the causal hierarchy —association— is considered to be purely statistical.
  The expressivity of iSPN stems from both the capacities of gate function and the knowledge of intervention as well as availability of respective data. As an important remark, causal inference is often interested in estimating interventional distributions, i.e, causal quantities from purely observational models. Therefore, an alternative formulation to Prop. 1 would be to replace the knowledge of the intervened causal structure G with knowledge on a valid adjustment set. In the following, we only consider the direct setting where actual interventional data from the system is assumed to be captured7 thereby freeing the investigation of iSPN from the independent research around hidden confounding.
  Universal Function Approximation (UFA). The gating nodes of CSPNs extend SPNs in a way that allows them to also induce functions which are universal approximators. For instance, using threshold gates xi ≤ c ∈ R, one can realize testing arithmetic circuits [Choi and Darwiche, 2018] which have been proven to be universal approximators - rendering iSPN to be UFA by construction.
  Learning of iSPN. An interventional sum-product network is being learned using a set of mixeddistribution samples generated from simulating the Causal Health SCM for different interventions, where the observational case is considered to be equivalent to an intervention on the empty set. The parameters θ,ψ of the iSPN describe the weights of the gate nodes and the distributions at the leaf nodes. The full model m is differentiable if the provided gate function f and each of the leaf models of g are differentiable. Therefore, to train an iSPN, as depicted in Fig. 2(b), it is sufficient to optimize the conditional log-likelihood end-to-end using gradient based optimization techniques. We assess the performance of our learned model through inspection of the adaptation of the model to the different interventions manifesting in the resulting marginals p(Vi | do(Uj)). As can be observed in Fig. 2(c) or alternatively in Fig. 4 (top row), the learned iSPN successfully adapts to both the interventions as well as its consequences. Considering for instance the intervention p(Vi | do(F = B( 12 ))) which removes the edge A→ F and thus renders Age (A) and Food Habits (F ) independent. Given the drastic population change in F and the fact that the Health (H) of an individual is causally dependent on both A and F a significant change in H is being expected. Indeed, both H and Mobility (M ), being a causal child of H , broaden distribution wise and also these subsequent changes are captured correctly.
  Causal Estimation. The algebraic-graphical do-calculus [Pearl, 2009] is complete in that it can find estimands for any identifiable query in a finite amount of transformation. An SPN, and thereby also iSPN, is capable of providing estimates to said demands when given observational data (L1 on the PCH). Consider for instance the well-known non-Markovian "Napkin" graph G = {W → Z → X → Y,W ∗↔ X,W ∗↔ Y } where ∗↔ denotes a confounded relation. The causal effect is identified as p(y| do(x)) = ( ∑ w p(x|w, z)p(w))−1( ∑ w p(y, x|w, z)p(w)) using the do-calculus where each of the r.h.s. components can be modelled by (i)SPN respectively. However, iSPN are also capable of directly expressing the causal quantity p(y| do(x)) as proven in Prop.1, similar to other neural-causal models like CausalGAN [Kocaoglu et al., 2019] or MLP-based NCM [Xia et al., 2021].
  Tractability. Inference in BNs and Markov networks is at least an NP-problem and existing exact algorithms have worst-case exponential complexity [Cooper, 1990, Roth, 1996], while SPNs can do inference in time proportional to the number of links in the graph. I.e., SPNs in general are able to compute any marginalization and conditioning query in time linear of the model’s representation size r, that is O(r). We deploy simple neural network architectures for which the runtime for the forward-pass is that of matrix-multiplication i.e., the time complexity scales cubically in the size of the input n, that is O(n3). The gradient descent procedure that involves forward and backward passes, assuming m gradient descent iterations, scales to O(mn3) which is the overall complexity we achieve during training phase. For SPN we deploy a random SPN structure which circumvents structure learning as such. Therefore, overall, training will generally be in the O(mn3r) regime while any causal query (that is, both L1 observational and L2 interventional in our case) will be answered within O(n3r). However, assuming that the weights of the random SPN were already initialized by the neural parameter-provider in a previous step, any causal query becomes answerable in O(r). Since asking for inferences/queries q within a single distribution (q → d ∈ Li) are more common than changes between distributions (d, d̂ ⊂ Li), this linear complexity by our tractable model is being leveraged fully for performing causal inference.
  Discussion. To reconsider and answer the general question of why the modelling of a conditional distribution via an over-parameterized architecture is a sensible idea consider the following. One can
  7For details on the remarked alternative formulation consider the Supplement.
  represent a conditional distribution p(Y | X) by applying Bayes Rule to a joint distribution density model (e.g. a regular SPN) p(Y | X) = p(Y,X)p(X) . However, this assumes non-empty support i.e., p(X) > 0. Furthermore, the joint distribution p(Y,X) optimizes all possibly derivable distributions, diminishing single distr. expressivity. Therefore, our considered formulation of a gate model allows for effectively subsuming the do-operator i.e., the gate model orchestrates the do queries such that the density estimator can easily switch between different interventional distributions. While not introducing specific limitations, general CSPN limitation regarding OOD generalization are inherited.
sections.heading: 4 Experimental Results
sections.text: The assumptions made in causality usually require control over the data generating process which is almost never readily available in the real world. This amounts to scarcity of the available public data sets and also their implications for transfers to the real world and even when available, they are usually artificially generated as some causal extension of a known, pre-existing data set (e.g. MorphoMNIST data set introduced by Castro et al. [2019]). While it is difficult to consider realworld-esque experimental settings for causal models, we do not restrict our investigations of iSPN to rather specific problem settings like certain noise or decision variable instances (which is common in causal inference). The evaluation is performed on data sets with varying number of variables and in both continuous and discrete domains. For the introduced causal health data set, we even consider arbitrary underlying noise distributions. We have made our code repository publicly available8.
  Data Sets. We evaluate iSPNs on four data sets. Three benchmarks: ASIA (A) [Lauritzen and Spiegelhalter, 1988] with 8 variables, Earthquake (E) with 5 variables and Cancer (C) [Korb and Nicholson, 2010] with 5 variables. One newly curated synthetic causal health (H) data set with 4 variables. More information about the data sets is presented in the Supplement.
  Baselines. For generative capacities, we compare our method against Mixture Density Networks (MDN) [Bishop, 1994] and Masked Autoencoder for Density Estimation (MADE) [Germain et al., 2015]. Both methods are expressive, parametric neural network based approaches for density estimation. Generally, the causality for machine learning literature suggests a strong favor for neural based function approximators for modelling causal mechanisms [Ke et al., 2019]. For causal capacities, we compare our method against the renown causal baselines from CausalML [Chen et al., 2020] and DoWhy [Sharma and Kiciman, 2020] for the modelling of average treatment effects (ATE) [Pearl, 2009, Peters et al., 2017] considered to be a gold standard task within causal inference.
  Protocol and Parameters. To account for reproducibility and stability of the presented results, we used learned models for five different random seeds per configuration. For means of visual clarity, the competing baselines MDN and MADE only present the best performing seed while iSPN is being presented with a mean plot and the standard deviation. The CBN, which performs exact inference according to the do-calculus (while best performing) has to be considered as gold standard and is therefore not part of the visualization. Furthermore, as it cannot compare in terms of feasibility. The considered interventions were of uniform nature. Each block of four columns represents a variable being randomized uniformly. We deployed a RAT-SPN [Peharz et al., 2020] selecting the leaf node distributions to be Gaussian distributions. For further experimental details consider the Supplement.
  8https://github.com/zecevic-matej/iSPN
  Our empirical analysis investigates iSPN for the following questions: Q1: How is the estimation quality for interv. distributions? Q2: How important is the model’s capacity? Q3: How is the runtime performance? Q4/5: How is the performance relative to state-of-the-art generative and causal models? Q6: How does the model adapt to different types of interventions?
  (Q1. Precise Estimation of do-influenced variables) The density functions learned by iSPN (see Fig. 4) fit with a high degree of precision as visualized by the difference in the peak of the modes of the learned distribution and that of the ground truth. While our visual analysis is arguably the superior method of evaluation as it conveys information about how the distributions of interest compare (which is feasible due to marginal inference and the locality of interventions within a SCM), we have also considered numerical evaluation criteria like the Jenson-Shannon-Divergence on which (as visually confirmed) iSPN outperforms the baselines (see Tab.1). For more detailled observation and interpretation consider the Supplement.
  (Q2. Capacity Ablation Study) We test the robustness of iSPN as the size of the associated SPN g(D;ψ) is varied. We obtain 5 different iSPNs for each of the 4 data sets by using 5 different numbers of sum node weights, 600, 1200, 1800, 2400, 3200, effectively changing the capacity of the parameter-sharing neural network f(G;θ). We observe iSPNs to be robust to varying hyperparameters that control the size of the SPN g(·) and effectively the complexity of the associated function approximator f(·). More details and figures are in the Supplement. (Q3. Comparison of running times) SPNs are tractable by design as long the networks size is polynomial in the input size. The superiority in running time also becomes apparent during training on the same (Causal Health) data set where we observe the mean run times over 50 passes of the whole data set to be significantly faster than competing methods (see Fig.3).
  (Q4. Comparison to Generative models) We compare the performances in terms of precision of fit of the learned distributions as well as the flexibility of the models. iSPN outperforms the baselines across all 4 data sets both in precision of the fit of the learned density functions and flexibility of adaptation to the different interventions as seen in figure 4. In our experiments, MDN had the worst performance with estimated densities being consistently and significantly different from the ground truth, as the model settled for an average distribution across all interventions. MADE is able to estimate to high precisions but showed to be generally inconsistent across the experimental settings.
  (Q5. Comparison to Causal models) We compare the numerical results as seen in figure 5 and observe that iSPN matches the performance of the causal baselines. The simple regressor employed by CausalML fails in the confounding case as it estimates wrongly the conditional, both DoWhy and iSPN can handle even the more difficult Simpson’s paradox [Simpson, 1951] scenario. An analytical derivation for the ATE is exampled in the Supplement.
  (Q6. Different Types of Intervention) By construction, iSPNs are capable to handle arbitrary interventions and our empirical results corroborate this impression. Fig.6 shows the training of multiple models on different interventions alongside two example interventions and their appearance in the marginal distributions. Depending on the training setup, e.g. how many different interventional distributions need be learned, any single intervention might become more easily optimizable since the model can exploit similarities between distributions. For a more detailled elaboration and also visualizations of other interventional distributions (including atomic and non-continuous interventions), we direct the reader to our extensive supplementary material.
sections.heading: 5 Conclusions
sections.text: We presented a way to connect causality with tractable probabilistic models by using sum-product networks parameterized by universal function approximators in the form of neural networks. We show that our proposed method can adapt to the underlying causal changes in a given domain and generate near perfect interventional distributions irrespective of the data distribution and the intervention type thereby exhibiting flexibility. Our empirical evaluation shows that our method is able to precisely estimate the conditioned variables and outperform generative baselines.
  Finding a different compilation method for SPNs such as by making use of tree CBNs is important for learning pure causal probabilistic models. Testing our method on larger real world causal data sets is an interesting direction. Finally, using rich expert domain knowledge in addition to observational data is essential for causality and extending our method to incorporate such knowledge is essential.
  Acknowledgments and Disclosure of Funding
  The authors thank the anonymous reviewers for their valuable feedback. This work was supported by the ICT-48 Network of AI Research Excellence Center “TAILOR" (EU Horizon 2020, GA No 952215) and by the Federal Ministry of Education and Research (BMBF; project “PlexPlain”, FKZ 01IS19081). It benefited from the Hessian research priority programme LOEWE within the project WhiteBox, the HMWK cluster project “The Third Wave of AI.” and the Collaboration Lab “AI in Construction” (AICO) of the TU Darmstadt and HOCHTIEF. Furthermore, the authors gratefully acknowledge the support of 1R01HD101246 from NICHD and W911NF2010224 from US Army Research Office (ARO).
title: Interventional Sum-Product Networks: Causal Inference with Tractable Probabilistic Models