DeepProbLog: Neural Probabilistic Logic Programming
We introduce DeepProbLog, a probabilistic logic programming language that incorporates deep learning by means of neural predicates. We show how existing inference and learning techniques can be adapted for the new language. Our experiments demonstrate that DeepProbLog supports (i) both symbolic and subsymbolic representations and inference, (ii) program induction, (iii) probabilistic (logic) programming, and (iv) (deep) learning from examples. To the best of our knowledge, this work is the first to propose a framework where general-purpose neural networks and expressive probabilistic-logical modeling and reasoning are integrated in a way that exploits the full expressiveness and strengths of both worlds and can be trained end-to-end based on examples.
The paper proposes DeepProbLog, a combination of probabilistic logic programming (PLP) and neural networks. In PLP, each atom is a Boolean random variable. The integration is achieved by considering NN that output a probability distribution, such as those that have a final softmax layer. In that case, the output of a network can be associated to a PLP construct such as a probabilistic fact or annotated disjunction and, in turn, to ground facts for so called neural predicates. Inference can then be performed by computing the probability of the atoms for neural predicates by forward evaluation of the network plus knowledge compilation to sentential decision diagrams. Learning is performed by gradient descent using algebraic ProbLog and the gradient semiring. The computation of the gradient is performed by a dynamic programming algorithm that also computes the gradient of the output wrt to the neural network output, so that the gradient can be propagated to the network for its training. DeepProbLog is applied to problems combining logical and perceptual reasoning such as computing the addition of two numbers given as images, inducing programs and probabilistic programming problems. The results of the comparison with CNN for addition and differentiable Forth for program induction shows that it converges faster to a better solution.  The integration of logic and NN is a hot topic nowadays. Among the many proposal, I find this one particularly interesting for its simplicity that makes it a good candidate to become a reference for future systems. The idea of associating the output of NNs to the probability of being true of ground atoms makes the integration direct and clean. Then inference and learning are obtained by propagating probabilities and gradients through the circuit and network. The integration is made possible by the use of probabilistic logic programming that attaches real values to ground atoms.  DeepProbLog is correctly placed in the context of related work. In particular, Rockta¨schel and Riedel (2017) integrates logic and NN by embedding logical symbols and modifying theorem proving. Learning then consists of tuning the embeddings. DeepProbLog differs because logical reasoning is retained with its soundness guarantees. Cohen et al. (2018) is more similar to DeepProbLog but uses a language that does not have a sound probabilistic semantics.  So the paper falls in a very active area but presents an approach that is nevertheless novel and promising. The experiments are particularly interesting because they show the effectiveness of DeepProbLog and because they can form benchmarks for systems aiming to integrate logical and perceptual reasoning.  The paper is overall clear and well written, with a balanced presentation that is easy to follow. I have only a remark on the example in section 4, which contains some imprecisions: 1) in the text you say the program has 6 parameters, yet nodes in Figure 2c have vectors with 5 or 6 elements 2) given that 3 of the 6 parameters are fixed given the other 3, why didn't you use just 3 parameters? 3) what is the meaning of p(0.5) in Figure 2b? 4) You say: "According to the neural network, the first coin is most likely heads (p = 0.9), but is entirely unsure about the  second (p = 0.5)." However, Fig 2c shows p=0.2 for side(coin2,heads)  After reading the rebuttal, I think the authors answered sufficiently well the reviewers' concerns.