## Conditional independence testing: a predictive perspective

Marco Henrique de Almeida Inácio · Rafael Izbicki Rafael Bassi Stern

·

Abstract Conditional independence testing is a key problem required by many machine learning and statistics tools. In particular, it is one way of evaluating the usefulness of some features on a supervised prediction problem. We propose a novel conditional independence test in a predictive setting, and show that it achieves better power than competing approaches in several settings. Our approach consists in deriving a p-value using a permutation test where the predictive power using the unpermuted dataset is compared with the predictive power of using dataset where the feature(s) of interest are permuted. We conclude that the method achives sensible results on simulated and real datasets.

Keywords Supervised learning · Conditional independence testing · Hypothesis test

## 1 Introduction

Conditional independence testing is a key problem required by many machine learning and statistics tools, including Bayesian networks (Jensen 1996; Campos 2006), time series (Diks and Panchenko 2006) , causal inference (Spirtes et al. 2000; Pearl 2009) and feature selection (Koller and Sahami 1996).

Unfortunately, it is not possible to design conditional independence tests that are powerful against all points in the alternative hypothesis (Shah and Peters 2018). Nevertheless, this issue can be partially addressed by assuming additional structure on the data distribution. Indeed, various conditional independence methods take advantage of specific settings to obtain improved power for alternatives of interest; see, for instance, Doran et al. (2014), Sen et al. (2017), Berrett et al. (2018), and Chalupka, Perona, and Eberhardt (2018) and references therein.

In this work we are interested in testing conditional independence as a way of evaluating the usefulness of some features on a supervised prediction problem. More precisely, let X S ⊂ ( X ,. . . , X 1 p ) be a subset of the features. Our goal is to test if X S is independent of the label Y conditionally on X S c , the remaining variables, i.e, we wish to test the hypothesis H 0 : X S ⊥ Y | X S c .

This is setting is closely connected to the literature of designing effective variable importance measures , in which the goal is to design indices that can be used to rank features according to how useful they are for predicting Y . A popular measure of variable importance was designed by Breiman (2001), but several alternative procedures are also available; see, for instance, Strobl et al. (2008) and Fisher, Rudin, and Dominici (2018) and references therein. Conditional independence testing is distinct from designing importance measures in the sense that its goal is not to quantify how informative a given feature is, but instead to answer the question: 'is this feature relevant?".

In this work we propose an approach to answer this question that consists in comparing the performance of two prediction methods: the first is trained using all features, while the second uses noise instead of the variables S . We show that our method yields a formal statistical hypothesis that approximately controls the significance level, and that it achieves considerable power against relevant alternative hypotheses.

Our work is related to Watson and Wright (2019), who also compare the risk of two prediction methods in order to test H 0 . The main difference between these methods is that we use a permutation test-based statistic to compute p-values (see Section 3 for further details). We show that this leads to substantial gain of power in several settings, and also a better control of type I error probabilities, especially for smaller sample sizes.

The remaining of this paper is organized as follows. Section 2 introduces COINP, our approach to test conditional independence. Section 3 contains experiments for comparing COINP with other approaches while

M. Inácio

Department of Statistics - Federal University of SÃčo Carlos (UFSCar) and

Institute of Mathematics and Computer Sciences - University of SÃčo Paulo (USP)

E-mail: m@marcoinacio.com

R. Izbicki

Department of Statistics - Federal University of SÃčo Carlos (UFSCar)

R. Stern

Department of Statistics - Federal University of SÃčo Carlos (UFSCar)

Section 4 presents an illustrative example of applying the method to a real world dataset together the classical importance measure obtained from random forests. Section 5 concludes the paper with final remarks.

## 2 Conditional Independence Predictive Test (COINP)

## 2.1 Notation and problem setting

Let X denote the feature space and Y the label space. The observed data is Z = ( X , Y ), where X ∈ X n is a n × p feature matrix and Y ∈ Y n is the label vetor. We assume that the observations z i = ( x i , y i ), i = 1 , . . . , n , are independent and identically distributed. Let X S ⊂ ( X ,. . . , X 1 p ) be a subset of the features. Our goal is to test if X S is independent of Y conditionally on X S c , the remaining variables, i.e, we wish to test the hypothesis H 0 : X S ⊥ Y | X S c .

We denote by F := { f : X -→ Y} the space of all mappings from features to outcomes (i.e., all prediction functions), and by Z the set of all datasets. A prediction method (e.g., neural networks or random forests) is a function in the space A = { a : Z -→ F} . We denote a loss function by L : Y × Y -→ R . The risk of a prediction function f ∈ F is R f ( ) := E [ L f ( ( X ) , Y )]. We denote by

$$\widehat { R } ( f, \tilde { \mathbb { Z } } ) \coloneqq \frac { 1 } { n } \sum _ { i = 1 } ^ { n } L ( f ( \tilde { \mathbf x } _ { i } ), \tilde { y } _ { i } )$$

the estimate of the risk of f ∈ F based on a holdout dataset ˜ of size Z n (i.e., a dataset not used for obtaining f ).

The Conditional Independence Predictive Test (COINP) requires one to randomly permuting the rows of X associated to the features S . This procedure is illustrated in Figure 1 for an example with S = { X 3 } . For every j = 1 , . . . , B , denote by Z π j the j -th dataset obtained by performing this procedure on Z , and let x π j i be the i -th observation of such dataset. Similarly, we denote by ˜ Z π j the j -th permutation of the holdout set ˜ . Z Table 1 summarizes the notation used in this paper.

Fig. 1: Permutation procedure used to test if the X 3 is independent of Y conditionally on X 1 and X 2 .

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00105v1-figure-8b117c26-3158-4670-af8b-f06854bc7086.jpg)

Table 1: Notation used in the paper.

| Symbol      | Meaning                                                                 |
|-------------|-------------------------------------------------------------------------|
| X           | n × p feature matrix                                                    |
| Y           | Label vector                                                            |
| Z           | Training data                                                           |
| ˜ Z         | Holdout data                                                            |
| F           | space of all prediction functions                                       |
| A           | space of all prediction methods                                         |
| ̂ R ( f, Z ) | Estimate of the risk of f ∈ F based on Z                                |
| Z π j       | j -th dataset with the rows of the S columns of Z are randomly permuted |
| x π j i     | The features of the i -th row of Z π j                                  |

## 2.2 Proposed method

The COINP procedure consists in testing H 0 by computing the rank of ̂ R a ( ( Z ) , ˜ ) among Z

$$\{ \widehat { R } ( a ( \mathbb { Z } ^ { \pi _ { 1 } } ), \tilde { \mathbb { Z } } ^ { \pi _ { 1 } } ), \dots, \widehat { R } ( a ( \mathbb { Z } ^ { \pi _ { B } } ), \tilde { \mathbb { Z } } ^ { \pi _ { B } } ) \},$$

where a ∈ A is any prediction method. We then reject the null hypothesis if this statistic is small. The intuition of this procedure is that, if H 0 does not hold (i.e., if the features S still have information about Y even given S c ), an algorithm that uses S in addition to S c will result in greater predictive power than one that does not use those features. Thus, permuting the rows associated with S will result in a prediction function with larger risk. Hence, if H 0 does not hold, ̂ R a ( ( Z ) , ˜ ) should be smaller than most Z ̂ R a ( ( Z π j ) , ˜ Z π j )'s. On the other hand, if H 0 holds, the features X S bring no additional gain in the predictive performance of a . It follows that the estimated risk ̂ R a ( ( Z ) , ˜ ) should be identically distributed to Z ̂ R a ( ( Z π j ) , ˜ Z π j ). It follows that the rank statistic is uniformly distributed under the null hypothesis. This justifies the following COINP procedure, which we formally state in the following definition.

Definition 1 (COINP - Conditional Independence Predictive Test) Let α ∈ (0 , 1) and a ∈ A be a predictive method. The α -level Conditional Independence Predictive Test consists in rejecting the null hypothesis H 0 : X S ⊥ Y | X S c if, and only if,

$$\frac { 1 } { B } \sum _ { j = 1 } ^ { B } \mathbb { I } \left ( \widehat { R } ( a ( \mathbb { Z } ), \tilde { \mathbb { Z } } ) \geq \widehat { R } ( a ( \mathbb { Z } ^ { \pi _ { j } } ), \tilde { \mathbb { Z } } ^ { \pi _ { j } } ) \right ) \leq \alpha.$$

The COINP is described in Algorithm 1. Notice that the left-hand side of Equation 1 is in fact a p-value based on a permutation test (Good 2013) .

## Algorithm 1 COINP

Input: training data Z , testing data ˜ , Z prediction method a ∈ A , loss function L , feature indices S , number of simulations B

Output: p-value for testing H 0 : X S ⊥ Y | X S c

- 1: f ← a ( Z )
- 2: R ← ̂ R f, ( ˜ ) Z
- 3: for j ∈ { 1 , . . . , B } do
- 4: Compute Z π j by randomly permuting the columns of Z associated to features S
- 5: f j ← a ( Z π j )
- 6: Compute ˜ Z π j by randomly permuting the columns of ˜ associated to features Z S
- 7: R j ← ̂ R f , ( j ˜ Z π j )
- 8: end for
- 9: return |{ j : R ≥ R j }| /B

## 3 Experiments

## 3.1 Other approaches

We compare our permutation method with the following approaches.

## 3.1.1 Conditional Predictive Impact (CPI)

The Conditional Predictive Impact (CPI) test, introduced by Watson and Wright (2019), consists in training two prediction methods: one on the original dataset, f = a ( Z ), and another one on a permuted dataset, f 1 = a ( Z π 1 ). It then tests the one-sided null hypothesis H 0 : R f, ( ˜ ) Z ≥ R f , ( 1 ˜ Z π 1 ) by checking if the distribution of the loss function on the original set,

$$( L ( f ( \tilde { \mathbf x } _ { 1 } ), \tilde { y } _ { 1 } ), \dots, L ( f ( \tilde { \mathbf x } _ { n } ), \tilde { y } _ { n } ) )$$

comes from a distribution with smaller average than the distribution of the loss function on the permuted test,

$$( L ( f _ { 1 } ( \tilde { \mathbf x } _ { 1 } ^ { \pi _ { 1 } } ), \tilde { y } _ { 1 } ), \dots, L ( f _ { 1 } ( \tilde { \mathbf x } _ { n } ^ { \pi _ { 1 } } ), \tilde { y } _ { n } ) ).$$

In practice, we use the paired t-test to perform this comparison; see Watson and Wright (2019) for other approaches.

## 3.1.2 Approximate CPI

We include a variation of CPI in which the same prediction function, f = a ( Z ), is used compute on both datasets. That is, in this version of the test, a paired t-test is used to compare the samples

$$( L ( f ( \tilde { \mathbf x } _ { 1 } ), \tilde { y } _ { 1 } ), \dots, L ( f ( \tilde { \mathbf x } _ { n } ), \tilde { y } _ { n } ) )$$

and

$$( L ( f ( \tilde { \mathbf x } _ { 1 } ^ { \pi _ { 1 } } ), \tilde { y } _ { 1 } ), \dots, L ( f ( \tilde { \mathbf x } _ { n } ^ { \pi _ { 1 } } ), \tilde { y } _ { n } ) ).$$

This procedure is essentially the method described by Breiman and Cutler (2008) to obtain p-values for the importance measures produced by random forests, with the exception that Breiman and Cutler (2008) use a z -test instead.

## 3.1.3 Approximate COINP

A drawback of COINP is that it is computationally intensive, especially if a is a slow predictive algorithm. We attempt to overcome these issues by computing the rank of ̂ R a ( ( Z ) , ˜ ) on Z

$$\{ \widehat { R } ( a ( \mathbb { Z } ), \tilde { \mathbb { Z } } ^ { \pi _ { 1 } } ), \dots, \widehat { R } ( a ( \mathbb { Z } ), \tilde { \mathbb { Z } } ^ { \pi _ { B } } ) \},$$

that is, we train a only once (on the original dataset). In other words, approximate COINP using the same procedure as that described in Algorithm 1, with the exception that line 4 and 5 are now replaced by f j ← f .

## 3.2 Simulation study description

Next, we describe the details of the simulation study performed to evaluate the proposed method.

## 3.2.1 Simulation scenarios

We generate artificial datasets for our simulation study using various distributions. We restrict our comparisons to regression settings with the squared loss, L y, y ( ̂ ) = ( y -̂ y ) 2 , even though our method is general and can be used for classification as well. Moreover, in what follows we will always test conditional independence of a single feature, i.e., | S | = 1.

The first and second scenarios have features that are independent of each other.

Distribution 1:

Distribution 2:

```
                                                                                                                                                                                                       
                                                                                                                                                                                                       

                                                                                                                                                                                                        }
                                                                                                                                                                                                      
	                                                                                                                                                                                                        |                                                                                                                                                                                                        \
                                                                                                                                                                                                       |
                                                                                                                                                                                                       \

                                                                                                                                                                                                       |

                                                                                                                                                                                                      

                                                                                                                                                                                                       }

                                                                                                                                                                                                       }                                                                                                                                                                                                        <?                                                                                                                                                                                                        </?                                                                                                                                                                                                      


                                                                                                                                                                                                       <!                                                                                                                                                                                                        +                                                                                                                                                                                                        -
```

In the other settings, we add correlation to the features (here SKN stands for skew normal distribution with location, scale and shape parameters respectively):

Distribution 3:

Distribution 4:

```
Distribution 3:                                                                                                                                                                                                         Y _ { i } = X _ { i } \beta + \epsilon }                                                                                                                                                                                                        y _ { i } = X _ { i } \beta + \epsilon }                                                                                                                                                                                      y _ { i } = X _ { i } \beta + \epsilon                                                                                                                                                                                                        x _ { i } \sim N ( 0, \Sigma )                                                                                                                                                                                                       
                                                                                                                                                                                                        z _ { i } \sim N ( 0, \Sigma )                                                                                                                                                                                          y _ { i } \sim beta ( 1, 1 )  for j \in \{ 1, 2 \}                                                                                                                                                                                                        \Sigma _ { 0, 0 } = 1                                                                                                                                                                                                        Z \sim N ( - 0. 5, 1 )                                                                                                                                                                                                       y ^ { i } \sim N ( - 0. 5, 1 )                                                                                                                                                                                             y _ { i } \sim N ( - 0. 5, 1 )                                                                                                                                                                                          y _ { i } \sim N ( - 0. 5, 0 )                                                                                                                                                                                                       x ^ { i } \sim N ( - 0. 5, 1 )                                                                                                                                                                                         y _ { i } \sim N ( - 0. 5, 1, 1 )                                                                                                                                                                                                     y _ { j } \sim N ( - 0. 5, 1 )                                                                                                                                                                                            y _ { i } \sim N ( - 0. 6, 1 )                                                                                                                                                                                                     \Served input : X                                                                                                                                                                                                       y ^ { i } \sim N ( - 0. 5, 1 )
```

For all the combinations described above, we vary β S in { 0 0 01 0 1 , . , . , 0 6 . } and the number of observations in { 1000 10000 , } . Notice that, in all settings, H 0 holds if, and only if, β S = 0. Moreover, as | β S | increases, the conditional dependency of Y on X S also increases.

For each setting we run 200 independent tests to estimate the power of each test. We set B We compare three choices for the prediction function a :

= 100.

- 1. Linear regression ,
- implemented using scikit-learn Python package (Pedregosa et al. 2011).
- 2. Feedforward neural networks . The specification of the network is as follows:
- - Optimizer : we work with the Adamax optimizer (Kingma and Ba 2014) and decrease its learning rate if improvement is seen on the validation loss for a considerable number of epochs.
- - Initialization : we used the initialization method proposed by Glorot and Bengio (2010).
- - Layer activation : we chose ELU (Clevert, Unterthiner, and Hochreiter 2015) as activation function.
- - Stop criterion : a 90%/10% split early stopping for small datasets and a higher split factor for larger datasets (increasing the proportion of training instances) and a patience of 50 epochs without improvement on the validation set.
- - Normalization and number of hidden layers : batch normalization, as proposed by Ioffe and Szegedy (2015), is used in this work in order to speed-up the training process, specially since our networks have 5 hidden layers with 100 neurons each.
- - Dropout : here we also make use of dropout which as proposed by Hinton et al. (2012).
- - Software :
- we have PyTorch as framework of choice which works with automatic differentiation.
- 3. Random Forests . We use Python's scikit-learn package with all its default tuning parameters, except for n\_estimators (number of trees), which is increased to 300 for better prediction performance.

The Python package and implementation scripts for this work are available at: https://github.com/ randommm/nnperm .

## 3.3 Results

Figures 2 and 3 show the cumulative distribution functions of the p-values for each of the settings under the null hypothesis (i.e., for the choice β S = 0). Proper p-values need to be uniformly distributed under the null hypothesis, and thus the cumulative distribution function should be close to the 45 0 line. The figure indicates that approximate methods only lead to proper p-values in settings 1 and 2. These are exactly the cases in which the covariates are independent of each other . Moreover, the exact methods come closer to leading to proper p-values in most cases. Exceptions to this are p-values obtained by CPI using artificial neural networks. This possibly happens because the networks do not converge in some simulations. This leads to extremely large values for the loss functions in some cases, which directly influence the t-test used by CPI. COINP, on the other hand, is immune to outliers because it relies on the evaluation of ranks as opposed to averages. This in turn guarantees that the distribution of its p-values are closer to uniformity under a larger variety of settings. We notice that an attempt to get better results for CPI in these settings is to consider the logarithm of the loss function, as suggested by Watson and Wright (2019).

Next, we compare the power function of the testing methods. Because only COINP and CPI had valid p-values, we restrict the comparisons to these methods. Figures 4 and 5 show the power of each test as a function of β S . The plot indicates that all procedures achieve higher power as β S increases. Moreover, in most settings COINP leads to better power than CPI. In these examples, higher power is achieved when using a linear regression for COINP. This can be explain by the fact that in all settings the true nature of the conditional distribution of Y | x is close to linear. By comparing the COINP results from Figures 4 and 5, it is also clear the for larger sample sizes (Figure 5), the power of COINP is larger. This indicates that the testing procedure is consistent.

Fig. 2: Cumulative distribution function of the p-values for the setting with n = 1 000 , under the null hypothesis (i.e., for choice the β S = 0 ).

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00105v1-figure-48c0b7ca-da74-476c-bde2-332d6b833469.jpg)

Fig. 3: Cumulative distribution function of the p-values for the setting with n = 10 000 , under the null hypothesis (i.e., for choice the β S = 0 ).

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00105v1-figure-86d82ad3-5eaa-4f60-b5a1-e62fd7781941.jpg)

Fig. 4: Power function for all setting with n = 1 000 , for α = 5% .

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00105v1-figure-630ee03c-d684-40cd-917c-cbf230438976.jpg)

Fig. 5: Power function for all setting with n = 10 000 , for α = 5% .

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00105v1-figure-62e8cf17-8a9b-4d36-bda5-e5edecb63e2d.jpg)

## 4 A dataset analysis example

As an illustrative example, we take the classical diamonds dataset which is readily available from ggplot2 library and Kaggle. We work with price as the response variable.

In Table 2, we present the p-values for each model together with the classical importance measure obtained from random forests.

Table 2: P-values for hypothesis testing for each comparison method compared to Random forest traditional importance measures.

|                |                |   carat |   depth |   table |    x |    y |    z |   cut |   color |   clarity |
|----------------|----------------|---------|---------|---------|------|------|------|-------|---------|-----------|
| ann            | App COINP      |    0    |    0    |    0    | 0    | 0    | 0    |  0    |    0    |      0    |
| ann            | App CPI        |    0    |    0.02 |    0.03 | 0    | 0    | 0    |  0    |    0    |      0    |
| ann            | COINP          |    0    |    0.6  |    0.99 | 0.06 | 0.38 | 0.78 |  0.01 |    0    |      0    |
| ann            | CPI            |    0.01 |    1    |    0    | 0    | 1    | 0.89 |  0.17 |    0    |      0    |
|                | App COINP      |    0    |    0    |    0    | 0    | 0.33 | 0.89 |  0    |    0    |      0    |
|                | App CPI        |    0    |    0.02 |    0    | 0    | 0.06 | 0.01 |  0    |    0    |      0    |
|                | COINP          |    0    |    0    |    0    | 0    | 0.27 | 0.92 |  0    |    0    |      0    |
|                | CPI            |    0    |    0.02 |    0.07 | 0    | 0    | 0.01 |  0.03 |    0    |      0    |
|                | App COINP      |    0    |    0    |    0    | 0    | 0    | 0    |  0    |    0    |      0    |
|                | App CPI        |    0    |    0    |    0.04 | 0    | 0    | 0    |  0    |    0    |      0    |
|                | COINP          |    0    |    0.05 |    0    | 0.04 | 0    | 0.02 |  0    |    0    |      0    |
|                | CPI            |    0    |    0.38 |    0.03 | 0.17 | 0    | 0.19 |  0.34 |    0    |      0    |
| RF Imp measure | RF Imp measure |    0.6  |    0.01 |    0    | 0.01 | 0.28 | 0.01 |  0    |    0.03 |      0.06 |

## 5 Final remarks

We have developed a novel approach for testing conditional independence under a predictive setting. We have shown that the p-values obtained by our approach are proper, and that our hypothesis test has larger power than competing approaches under a variety of settings.

When compared to CPI, our approach is especially appealing for small sample sizes, because (i) it does not rely on asymptotic approximations such as those required by the t-test, and (ii) its computational burden is not high in those cases.

## Acknowledgments

Marco Henrique de Almeida Inacio is grateful for the financial support of CAPES: this study was financed in part by the CoordenaÃğÃčo de AperfeiÃğoamento de Pessoal de NÃŋvel Superior - Brasil (CAPES) Finance Code 001. Rafael Izbicki is grateful for the financial support of FAPESP (2017/03363-8) and CNPq (306943/2017-4).

## References

Berrett, Thomas B., et al. 2018. The conditional permutation test . eprint: arXiv:1807.05405 .

Breiman, L., and A. Cutler. 2008. 'Random Forests for scientific discovery'. Visited on 02/20/2019.

//www.stat.berkeley.edu/~breiman/RandomForests/ENAR\_files/frame.htm

.

Breiman, Leo. 2001. 'Random forests'. Machine learning 45 (1): 5-32.

Campos, Luis M de. 2006. 'A scoring function for learning Bayesian networks based on mutual information and conditional independence tests'. Journal of Machine Learning Research 7 (Oct): 2149-2187.

Chalupka, Krzysztof, Pietro Perona, and Frederick Eberhardt. 2018. 'Fast Conditional Independence Test for Vector Variables with Large Sample Sizes'. arXiv preprint arXiv:1804.02747 .

Clevert, Djork-ArnÃľ, Thomas Unterthiner, and Sepp Hochreiter. 2015. Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs) . eprint: arXiv:1511.07289 .

Diks, Cees, and Valentyn Panchenko. 2006. 'A new statistic and practical guidelines for nonparametric Granger causality testing'. Computing in economics and finance, Journal of Economic Dynamics and Control 30 (9): 1647-1669. issn : 0165-1889.

https:

Doran, G, et al. 2014. 'A permutation-based kernel conditional independence test' (): 132-141.

- Fisher, Aaron, Cynthia Rudin, and Francesca Dominici. 2018. 'All Models are Wrong but many are Useful: Variable Importance for Black-Box, Proprietary, or Misspecified Prediction Models, using Model Class Reliance'. arXiv preprint arXiv:1801.01489 .

Glorot, Xavier, and Yoshua Bengio. 2010. 'Understanding the difficulty of training deep feedforward neural networks', 9 (): 249-256.

- Good, Phillip. 2013. Permutation tests: a practical guide to resampling methods for testing hypotheses . Springer Science &amp; Business Media.

Hinton, Geoffrey E., et al. 2012. 'Improving neural networks by preventing co-adaptation of feature detectors'. CoRR .

Ioffe, Sergey, and Christian Szegedy. 2015. 'Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift'. In Proceedings of the 32nd International Conference on Machine Learning , ed. by Francis Bach and David Blei, 37:448-456. Proceedings of Machine Learning Research. Lille, France: PMLR.

Jensen, Finn V. 1996. An introduction to Bayesian networks . Vol. 210. UCL press London.

Kingma, Diederik P., and Jimmy Ba. 2014. 'Adam: A Method for Stochastic Optimization.' CoRR abs/1412.6980.

Koller, Daphne, and Mehran Sahami. 1996. 'Toward optimal feature selection'. In Proceedings of the Thirteenth International Conference on International Conference on Machine Learning , 284-292. Morgan Kaufmann Publishers Inc.

Pearl, Judea. 2009. Causality . Cambridge university press.

Pedregosa, F., et al. 2011. 'Scikit-learn: Machine Learning in Python'. Journal of Machine Learning Research 12:2825-2830.

Sen, Rajat, et al. 2017. 'Model-Powered Conditional Independence Test'. In Advances in Neural Information Processing Systems 30 , ed. by I. Guyon et al., 2951-2961. Curran Associates, Inc.

- Shah, Rajen D., and Jonas Peters. 2018. The Hardness of Conditional Independence Testing and the Generalised Covariance Measure . eprint: arXiv:1804.07203 .
- Spirtes, Peter, et al. 2000. Causation, prediction, and search . MIT press.
- Strobl, Carolin, et al. 2008. 'Conditional variable importance for random forests'. BMC bioinformatics 9 (1): 307.
- Watson, David S, and Marvin N Wright. 2019. 'Testing Conditional Predictive Independence in Supervised Learning Algorithms'. arXiv preprint arXiv:1901.09917 .
Marco Henrique de Almeida Inácio Rafael Izbicki Rafael Bassi Stern
Conditional independence testing: a predictive perspective
Conditional independence testing is a key problem required by many machine
learning and statistics tools. In particular, it is one way of evaluating the
usefulness of some features on a supervised prediction problem. We propose a
novel conditional independence test in a predictive setting, and show that it
achieves better power than competing approaches in several settings. Our
approach consists in deriving a p-value using a permutation test where the
predictive power using the unpermuted dataset is compared with the predictive
power of using dataset where the feature(s) of interest are permuted. We
conclude that the method achives sensible results on simulated and real
datasets.