Recent advances in Neural Variational Inference allowed for a renaissance in latent variable models in a variety of domains involving high-dimensional data. In this paper, we introduce two generic Variational Inference frameworks for generative models of Knowledge Graphs; Latent Fact Model and Latent Information Model. While traditional variational methods derive an analytical approximation for the intractable distribution over the latent variables, here we construct an inference network conditioned on the symbolic representation of entities and relation types in the Knowledge Graph, to provide the variational distributions. The new framework can create models able to discover underlying probabilistic semantics for the symbolic representation by utilising distributions which permit training by back-propagation in the context of neural variational inference, resulting in a highly-scalable method. Under a Bernoulli sampling framework, we provide an alternative justification for commonly used techniques in large-scale stochastic variational inference, which drastically reduces training time at a cost of an additional approximation to the variational lower bound. The generative frameworks are flexible enough to allow training under any prior distribution that permits application of the re-parametrisation trick, as well as under any scoring function that permits maximum likelihood estimation of the parameters. Experimental results display the potential and efficiency of this framework by improving upon multiple benchmarks with Gaussian prior representations. Code publicly available on Github.
The paper presents two variational inference frameworks for generative models of knowledge graphs. Such models are based respectively on latent fact model and latent information model.
The authors argue that with the presented framework the underlying probabilistic semantics can be discovered. Experiments show performances comparable with state-of-art approaches.
Unfortunately, the paper seems to me not clear and rather incomplete in its actual form.
Overall, the proposal is novel. I cannot decide about significance because results do not outperform those of other approaches. To this extent, the authors should better discuss the results, explaining in more detail why this approach should be used instead of others (scales better, is faster, etc.).
In the abstract, it is asserted that one can discover underlying probabilistic semantics, but in the corpus of the paper this aspect is not described or mentioned in detail.
Similar problem for the reference to von-Mises distribution. This distribution is just named, it is said that the framework can handle such a distribution, but a reference to a paper and/or a short paragraph to explain the sentence are missing. This statement now results to be just information disconnected by the rest of the paper.
In a similar way, many other points suffer from a poor organization in my opinion.
When describing LIM an error is introduced here that is then copied and pasted throughout the paper: in the productory on p, p is in R not in E. This is a simple typo, but the fact that it is repeated so many times, also in the proof, gives me the feeling that the paper was written at the last moment.
Figures 1 and 2 are never referred.
Formula 6 must be better explained. If I have not lost something, n is the number of labeled triples, s_c is undefined, b_c is the probability of s_c to be equal to 1, the index i is never used. The paper lacks information here.
As regards the experimental part, some results are shown in subsection 4.3 called link prediction, others in section 5 called link prediction analysis. This organization does not seem to me to be really optimal. I would suggest creating an experimental section.
Moreover, the tests should be better explained, the tables are shown without specifying how they are built and how the values are collected. Information is provided in the appendix but could be included in the paper as the maximum limit is of 10 pages (8 suggested but I think an extra half page can be used).
The knowledge bases used should be at least cited, I know that freebase and wordnet are well-known but somewhere, in the description of the test, the name should be included. Also to specify the characteristics of the versions (WN18 vs WN18RR). Moreover, what does the value -257 in column 1, row 4 means?
Then, it is said that Table 1 shows improvements for ComplEx, but such improvements are rather low, is there a way to prove their significance? Otherwise, I would say that the performance is the same for WN18.
Tables 1 and 2 contain cells with '-' value, what does it mean?
Discussion about table 3 is incomplete in my opinion. First of all, the "proportion" column should be described. Also, on one hand, it is true that the _member_meronym is the least accurate and prominent but the most problem may come from _hypernym, which is the most prominent and the accuracy is also low. This fact is highlighted for table 4 but not for table 3.
Minor issues
- sec 3: references to Miao et al. must be enclosed by brackets
- sec 4.3: "We believe this *is* due to ..."
- sec 5.2: what is Model A? Also, the sentence seems incomplete.
Pros:
- Novel approach
Cons
- Test results are not convincing
- The paper is not mathematically sound
- The paper needs to be re-organized