SMOOTHING THE GEOMETRY OF PROBABILISTIC BOX EMBEDDINGS
There is growing interest in geometrically-inspired embeddings for learning hierarchies, partial orders, and lattice structures, with natural applications to transitive relational data such as entailment graphs. Recent work has extended these ideas beyond deterministic hierarchies to probabilistically calibrated models, which enable learning from uncertain supervision and inferring soft-inclusions among concepts, while maintaining the geometric inductive bias of hierarchical embedding models. We build on the Box Lattice model of Vilnis et al. (2018), which showed promising results in modeling soft-inclusions through an overlapping hierarchy of sets, parameterized as high-dimensional hyperrectangles (boxes). However, the hard edges of the boxes present difficulties for standard gradient based optimization; that work employed a special surrogate function for the disjoint case, but we find this method to be fragile. In this work, we present a novel hierarchical embedding model, inspired by a relaxation of box embeddings into parameterized density functions using Gaussian convolutions over the boxes. Our approach provides an alternative surrogate to the original lattice measure that improves the robustness of optimization in the disjoint case, while also preserving the desirable properties with respect to the original lattice. We demonstrate increased or matching performance on WordNet hypernymy prediction, Flickr caption entailment and a MovieLens-based market basket dataset. We show especially marked improvements in the case of sparse data, where many conditional probabilities should be low, and thus boxes should be nearly disjoint.
This paper proposes a soft relaxation of the box lattice (BL) model of Vilnis et al. 2018 and applies it to several graph prediction tasks. Results are comparable to the BL model on existing artificially-balanced data but significantly better on more natural unbalanced data with a large number of negatives. The paper assumes some familiarity with the problem domain and existing works (there is not a lot of exposition for an unfamilar reader), but should be of strong interest to anyone working on embeddings or graph prediction.
The paper is well-written, with clear explanations of the desired properties of the model and a concise set of experiments that are easy to follow. The strongest result is that on unbalanced WordNet, while the Flickr and MovieLens results are a little less clear but do show that this technique does not cause any loss in performance.
A few points of feedback:
- Missing citation / comparison: https://arxiv.org/pdf/1804.01882.pdf (Ganea et al. 2018) is an alternative way of generalizing order embeddings. They also report very high numbers on WordNet, though I'm not sure they are directly comparable.
- The Gaussian relaxation (Eq. (2) and (3)) defines a particular length scale, \sigma. It's not clear if this is also implicit in the softplus derivation (by analogy with Eq. (4), should we assume that it approximates the \sigma = 1 case?). What effect does this have on the embedding space? Without it, it would seem that the normal BL model is scale invariant, which might be a desirable property for representing hierarchical data.
- The main thrust of section 5.2 is that smoothed box embeddings retain better performance with increasing numbers of negatives. Could you include the ratio of positive / negative examples on the Flickr dataset, and some measure of the distribution of P(A|B) values on MovieLens to get a sense of how these datasets compare?
- Flickr data: what is the encoder model that produces the embeddings here, and how does it handle unseen captions? (Why would we expect the smoothed box model to handle unseen captions better?)
- There's a strong emphasis on how smoothing makes training easier. Do you have any metrics to directly support this, such as variance under random restarts?
- In the abstract and introduction, it's easy to gloss over "inspired by" and assume that the actual model is a Gaussian convolution. Could be more direct here that it's a softplus approximation.