Markov Logic Networks (MLNs), which elegantly combine logic rules and probabilistic graphical models, can be used to address many knowledge graph problems. However, inference in MLN is computationally intensive, making the industrialscale application of MLN very difficult. In recent years, graph neural networks (GNNs) have emerged as efficient and effective tools for large-scale graph problems. Nevertheless, GNNs do not explicitly incorporate prior logic rules into the models, and may require many labeled examples for a target task. In this paper, we explore the combination of MLNs and GNNs, and use graph neural networks for variational inference in MLN. We propose a GNN variant, named ExpressGNN, which strikes a nice balance between the representation power and the simplicity of the model. Our extensive experiments on several benchmark datasets demonstrate that ExpressGNN leads to effective and efficient probabilistic logic reasoning.
The paper proposes to use graph neural networks (GNN) for inference in MLN. The main motivation seems to be that inference in traditional MLN is computationally inefficient. The paper is cryptic about precisely why this is the case. There is some allusion in the introduction as to grounding being exponential in the number of entities and the exponent being related to the number of variables in the clauses of the MLN but this should be more clearly stated (e.g., does inference being exponential in the number of entities hold for lifted BP?). In an effort to speed up inference, the authors propose to use GNN instead. Since GNN expressivity is limited, the authors propose to use entity specific embeddings to increase expressivity. The final ingredient is a mean-field approximation that helps break up the likelihood expression. Experiments are conducted on standard MLN benchmarks (UW-CSE, Kinship, Cora) and link prediction tasks. ExpressGNN achieves a 5-10X speedup compared to HL-MRF. On Cora HL-MRF seems to have run out of memory. On link prediction tasks, ExpressGNN seems to achieve better accuracy but this result is a bit difficult to appreciate since the ExpressGNN can't learn rules and the authors used NeuralLP to learn the rules followed by using ExpressGNN to learn parameters and inference.
Here are the various reasons that prevent me from rating the paper favorably:
- MLNs were proposed in 2006. Statistical relational learning is even older. This is not a paper where the related work section should be delegated to the appendix. The reader will want to know the state of inference and its computational complexity right at the very beginning. Otherwise, its very difficult to read the paper and appreciate the results.
- Recently, a number of papers have been tried to quantify the expressive power of GNNs. MLN is fairly general, being able to incorporate any clause in first-order logic. Does the combination with GNN result in any loss of expressivity? This question deserves an answer. If so, then the speedup isn't free and ExpressGNN would be a special case of MLN, albeit with the advantage of fast inference.
- Why doesn't the paper provide clear inference time complexities to help the reader appreciate the results? At the very least, the paper should provide clear time complexities for each of the baselines.
- There are cheaper incarnations of MLN that the authors should compare against (or provide clear reasons as to why this is not needed). Please see BoostSRL (Khot, T.; Natarajan, S.; Kersting, K.; and Shavlik, J. 2011. Learning Markov logic networks via functional gradient boosting. In ICDM)