We introduce Deep Reasoning Networks (DRNets), an end-to-end framework that combines deep learning with reasoning for solving pattern de-mixing problems, typically in an unsupervised or weakly-supervised setting. DRNets exploit problem structure and prior knowledge by tightly combining logic and constraint reasoning with stochastic-gradient-based neural network optimization. We illustrate the power of DRNets on de-mixing overlapping hand-written Sudokus (Multi-MNIST-Sudoku) and on a substantially more complex task in scientific discovery that concerns inferring crystal structures of materials from X-ray diffraction data (Crystal-Structure-Phase-Mapping). DRNets significantly outperform the state of the art and experts' capabilities on Crystal-Structure-Phase-Mapping, recovering more precise and physically meaningful crystal structures. On Multi-MNISTSudoku, DRNets perfectly recovered the mixed Sudokus' digits, with 100% digit accuracy, outperforming the supervised state-of-the-art MNIST de-mixing models.
This paper proposes a new encoder-decoder framework that combines prior knowledge-based regularization and constrained reconstruction for unsupervised and weakly-supervised classification in structure rich scenarios. This framework injects prior knowledge in the form of relaxed constraints that act as regularization during the training of the encoder network. Some of the constraints concern sets of training examples. In this case, the paper proposes corresponding sampling schemes. Three experiments demonstrate the efficacy of the model. The first is a synthetically created 4x4 Sudoku made of overlaid MNIST digits. The other two are based on predicting crystal structures from x-ray diffraction measurements. Here, the first experiment is on simulated data for the Al-Li-Fe oxide system, while the other is performed on real measurements for the Bi-Cu-V oxide system.
Overall, I believe that the proposed framework could be a significant contribution to the fields of representation learning and constrained optimization. However, the paper exhibits serious shortcomings, which require revision. 
First, the positive aspects of the paper:
•	The framework is simple yet ingenious. It makes intelligent use of constraints in the form of regularization to guide the training of the encoder. Furthermore, it enables the direct design of the latent representation through the use of (pre-trained) generative models for constrained reconstruction of data points.
•	The proposed entropy-based method for relaxation of discrete constraints is intuitive and potentially adaptable for further constraints.
•	The experiments presented in this paper are well chosen. They demonstrate the contribution of the model to both general CV data as well as a specialized domain, where it can solve both simulated and real scenarios.
•	The paper provides an extensive literature survey, which makes it easy to embed the presented work in the proper context. However, I propose to remove the paragraph titled “Other less closely related work” as the connection to the current work is not clear, and the space could be used more effectively (see below).
Unfortunately, this paper has a couple of major flaws:
•	The results for DRNets (Generalization) on the MNIST Sudoku are compromised because the model trained on the test set for 25 epochs after being trained on the training set. Honestly, I was baffled to read the following sentence in the appendix: “Note that, during the test, instead of predicting the overlapping digits directly as other networks, we further optimize DRNets on the test set for 25 epochs to achieve a better result.” What is more, the main paper does not even mention this fact!
•	Although this paper relies on empirical verification of its proposition, the experimental results are almost impossible to interpret with just the information provided in the paper. Both experiments are poorly described, and even after reading the appendix several times, some serious detective work was necessary to piece together what happened in the experiments. The XRD experiments are especially hard to decipher, even with a physics background. Many vital components remain shrouded in mystery: What is a composition graph, and how are the paths sampled from it? How does the restart method, which is part of the results, work? Why are only six phases shown in the phase concentration visualizations if there were 159 possible phases? Are these the first six, a random subset, or were the other phases not realized?
•	The paper introduces the constraint-aware SGD algorithm to incorporate batching rules into the training of the encoder. On pages 2 and 6 and in Algorithm 1, I found the statement that the weights for each constrained are updated dynamically. However, that is where the information on the dynamic update method ends. Nowhere in the paper or the appendix did I find an explanation of how this is done. As this mechanism is a critical component of the proposed framework, the absence of an explanation is a significant oversight.
Other remarks:
•	In the context of global constraints, the paper talks about a constraint graph. If I understand the creation of this graph correctly, this graph has several connected components in which every element connects to every other element. As such, this seems to be a collection of sets rather than a real graph. This is especially confusing in the case of XRD, where all data points are in the same global constraint, leading to a fully connected “graph”.
•	Although I appreciate the reference to Kahneman’s model of the mind, I suggest to remove the first two paragraphs from the introduction and use the space to motivate the de-mixing problem instead. While it is a compelling (but not novel) observation, the analogy to system 1 and system 2 does not benefit the proposed work in the slightest.
•	In general, I fail to see the connection between reasoning and the proposed work. The model itself is an encoder-decoder network that cannot reason. It does not discover any new rules during training. All the reasoning has to be done manually beforehand to be then incorporated in the form of constraints. To clarify, I do believe that there is value in the presented work, but not necessarily in the way, it is advertised.