Perfect Match: A Simple Method for Learning Representations For Counterfactual Inference With Neural Networks
Learning representations for counterfactual inference from observational data is of high practical relevance for many domains, such as healthcare, public policy and economics. Counterfactual inference enables one to answer "What if...?" questions, such as "What would be the outcome if we gave this patient treatment ?". However, current methods for training neural networks for counterfactual inference on observational data are either overly complex, limited to settings with only two available treatment options, or both. Here, we present Perfect Match (PM), a method for training neural networks for counterfactual inference that is easy to implement, compatible with any architecture, does not add computational complexity or hyperparameters, and extends to any number of treatments. PM is based on the idea of augmenting samples within a minibatch with their propensity-matched nearest neighbours. Our experiments demonstrate that PM outperforms a number of more complex state-of-the-art methods in inferring counterfactual outcomes across several real-world and semi-synthetic datasets.
 This paper proposes an augmentation of traditional neural network learning to allow for the inference of causal effects. Specifically, they modify the data sampling procedure of SGD during training to use matched samples that are paired via propensity score matching. Experimental results on a number of dataset show that the proposed methodology is comparable to alternative machine learning based causal inference methods. Overall, I think this is a nice idea. I have two main concerns: (1) The use of small batches for matching. Figure 2 does alleviate this concern to an extent, but there is a large literature in statistics and the social sciences on the effect that the quality of matches have on the final causal estimand. It is quite possible that this particular dataset is more amenable to PSM. It is also worth noting that while there is bias reduction shown in figure 2, it is not overwhelming. (2) The use of propensity scores for matching. One of the insights from the heterogeneous treatment effect literature is that it is not difficult to find cases where the propensity of treatment is identical for two sets of covariates that otherwise do not obey any real balance. This can lead to large biases in the final estimate. Given that PSM is still a relatively widely used practice, I don’t think that its use is a ground for rejection in itself, but given that neural networks are often used to estimate complex causal relations when they are used and this paper is interested in individual treatment effects it is worth noting. I found the experimental setup to do a very good job in covering large portions of the behavior of the algorithm. The final results are a little underwhelming–the proposed method does not appear to clearly define a new state of the art for the tasks it is applied to–but it is often competitive and the paper presents an interesting idea.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Perfect Match: A Simple Method for Learning Representations For Counterfactual Inference With Neural Networks
Learning representations for counterfactual inference from observational data is of high practical relevance for many domains, such as healthcare, public policy and economics. Counterfactual inference enables one to answer "What if...?" questions, such as "What would be the outcome if we gave this patient treatment ?". However, current methods for training neural networks for counterfactual inference on observational data are either overly complex, limited to settings with only two available treatment options, or both. Here, we present Perfect Match (PM), a method for training neural networks for counterfactual inference that is easy to implement, compatible with any architecture, does not add computational complexity or hyperparameters, and extends to any number of treatments. PM is based on the idea of augmenting samples within a minibatch with their propensity-matched nearest neighbours. Our experiments demonstrate that PM outperforms a number of more complex state-of-the-art methods in inferring counterfactual outcomes across several real-world and semi-synthetic datasets.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 This paper proposes an augmentation of traditional neural network learning to allow for the inference of causal effects. Specifically, they modify the data sampling procedure of SGD during training to use matched samples that are paired via propensity score matching. Experimental results on a number of dataset show that the proposed methodology is comparable to alternative machine learning based causal inference methods. Overall, I think this is a nice idea. I have two main concerns: (1) The use of small batches for matching. Figure 2 does alleviate this concern to an extent, but there is a large literature in statistics and the social sciences on the effect that the quality of matches have on the final causal estimand. It is quite possible that this particular dataset is more amenable to PSM. It is also worth noting that while there is bias reduction shown in figure 2, it is not overwhelming. (2) The use of propensity scores for matching. One of the insights from the heterogeneous treatment effect literature is that it is not difficult to find cases where the propensity of treatment is identical for two sets of covariates that otherwise do not obey any real balance. This can lead to large biases in the final estimate. Given that PSM is still a relatively widely used practice, I don’t think that its use is a ground for rejection in itself, but given that neural networks are often used to estimate complex causal relations when they are used and this paper is interested in individual treatment effects it is worth noting. I found the experimental setup to do a very good job in covering large portions of the behavior of the algorithm. The final results are a little underwhelming–the proposed method does not appear to clearly define a new state of the art for the tasks it is applied to–but it is often competitive and the paper presents an interesting idea.
assistant
 This paper proposes an augmentation of traditional neural network learning to allow for the inference of causal effects. Specifically, they modify the data sampling procedure of SGD during training to use matched samples that are paired via propensity score matching. Experimental results on a number of dataset show that the proposed methodology is comparable to alternative machine learning based causal inference methods. Overall, I think this is a nice idea. I have two main concerns: (1) The use of small batches for matching. Figure 2 does alleviate this concern to an extent, but there is a large literature in statistics and the social sciences on the effect that the quality of matches have on the final causal estimand. It is quite possible that this particular dataset is more amenable to PSM. It is also worth noting that while there is bias reduction shown in figure 2, it is not overwhelming. (2) The use of propensity scores for matching. One of the insights from the heterogeneous treatment effect literature is that it is not difficult to find cases where the propensity of treatment is identical for two sets of covariates that otherwise do not obey any real balance. This can lead to large biases in the final estimate. Given that PSM is still a relatively widely used practice, I don’t think that its use is a ground for rejection in itself, but given that neural networks are often used to estimate complex causal relations when they are used and this paper is interested in individual treatment effects it is worth noting. I found the experimental setup to do a very good job in covering large portions of the behavior of the algorithm. The final results are a little underwhelming–the proposed method does not appear to clearly define a new state of the art for the tasks it is applied to–but it is often competitive and the paper presents an interesting idea.