Some fundamental limitations of deep learning have been exposed such as lacking generalizability and being vunerable to adversarial attack. Instead, researchers realize that causation is much more stable than association relationship in data. In this paper, we propose a new framework called do-calculus AutoEncoder(DO-AE) for deep representation learning that fully capture bivariate causal relationship in the images which allows us to intervene in images generation process. DO-AE consists of two key ingredients: causal relationship mining in images and intervention-enabling deep causal structured representation learning. The goal here is to learn deep representations that correspond to the concepts in the physical world as well as their causal structure. To verify the proposed method, we create a dataset named PHY2D, which contains abstract graphic description in accordance with the laws of physics. Our experiments demonstrate our method is able to correctly identify the bivariate causal relationship between concepts in images and the representation learned enables a do-calculus manipulation to images, which generates artificial images that might possibly break the physical law depending on where we intervene the causal system.
Thank you for your submission.
- What is the specific question/problem tackled by the paper?
The paper proposes a VAE architecture to learn causal relations and allow for interventions. The architecture requires knowledge of the causal graph, and the direction of the causal arrows are inferred by comparing the log-likelihoods of generated images. The architecture may also require knowledge that an arrow exists between two vertices. This relies on the principle that "low-capacity" neural networks can predict better along the causal arrows (with the cause as input and the effect as the output) than in the opposite direction (with the effect as input and the cause as the output).
The paper focuses on the graph (A, B) where one wants to understand whether A causes B, or B causes A. The paper also discusses intervening in this graph.
The paper uses a new dataset for evaluating the approach, based on simple Newtonian systems. 
- Is the approach well motivated, including being well-placed in the literature?
I think the motivation is adequate, but the review of the literature glosses over related work (or the absence thereof) in predicting the direction of arrows in causal graphs. The comparison of the proposed dataset against existing ones is missing.
- Does the paper support the claims? This includes determining if results, whether theoretical or empirical, are correct and if they are scientifically rigorous.
The procedure for determining whether A causes B (or B causes A) is qualitative. The paper demonstrates that the performance gap between the correct and incorrect explanations is consistently distinguishable across multiple experiments.
Visual inspection of the generated images is also used for assessing the quality of the models.
Because the results are qualitative, the support for the claims is not as strong as it could be (with quantitative results).
- Summarize what the paper claims to do/contribute. Be positive and generous.
The paper has two main contributions:
* Evidence to the Independent Mechanism principle (in a setting different from Bengio et al.'s transfer setup).
* A new dataset for evaluating learning causal arrows (with accessible ground-truths).
I think these are interesting contributions.
- Is the paper clearly written?
The paper has a number of grammatical errors that should be fixed.
The explanation of how the latent interventions are made is important and should be included.
- Clearly state your decision (accept or reject) with one or two key reasons for this choice.
I vote for a weak accept.
- Provide supporting arguments for the reasons for the decision.
I trust that the writing issues will be addressed in due course, but I am also concerned about the fact that evaluations are qualitative. The qualitative results provide support for the contributions that could be strengthened. 
The dataset is also an interesting contribution and it is a good idea to give it visibility. For this, though, it is important that the paper assess its strengths and limitations in comparison to alternative datasets.
- Provide additional feedback with the aim to improve the paper. Make it clear that these points are here to help, and not necessarily part of your decision assessment.
I am not convinced that mentioning Kolmogorov complexity is an efficient use of the space. I think the content could be improved by making the motivation section more concise and adding a few more experimental results or discussion.
Which discussions would be good to have? I think it should be noted that the intervention on effect should behave as demonstrated (creating implausible scenarios). Also some more development on the spring example: What is the right causal graph for it, and can the arrows in that graph be learned?
Quantitative results would also improve the paper. Maybe decide between A->B or A<-B based on a statistical test?
You give an example about elephant-grassland association. Please cite a source for that.
Suppose that both likelihoods for A->B and B<-A are about the same. How do you decide if your model is too rich, or if there's no relationship? (This is an important question to understand if the method requires knowledge that an (A,B) arrow exists or not.)
The panels in Figure 5 do not support the claim. The simple net gets better at the cause, but in some cases the rich representation does a better job at the effect.
I think the physics dataset is also a contribution, so its originality & impact should be discussed in comparison to related work. Why is this an adequate benchmark? How does it address limitations of other benchmarks that could be used to evaluate proposed solutions for the problem in question?
In summary, my suggestions for improving the paper are:
1) Make sure & demonstrate (by adequate discussion of related work) the originality of the contributions:
1.1) The method for detecting the direction of causal arrows.
1.2) The dataset as a benchmark for the problem being studied.
2) Report quantitative results across the dataset and maybe across multiple setups for each name/physical law, with good coverage. You may consider a test set where the parameters are within the sampling range of your training set, and also outside the sampling range (where success of the method would be even more interesting).