Workshop track -ICLR 2018 GEOSEQ2SEQ: INFORMATION GEOMETRIC SEQUENCE-TO-SEQUENCE NETWORKS
The Fisher information metric is an important foundation of information geometry, wherein it allows us to approximate the local geometry of a probability distribution. Recurrent neural networks such as the Sequence-to-Sequence (Seq2Seq) networks that have lately been used to yield state-of-the-art performance on speech translation or image captioning have so far ignored the geometry of the latent embedding, that they iteratively learn. We propose the information geometric Seq2Seq (GeoSeq2Seq) network which abridges the gap between deep recurrent neural networks and information geometry. Specifically, the latent embedding offered by a recurrent network is encoded as a Fisher kernel of a parametric Gaussian Mixture Model, a formalism common in computer vision. We utilise such a network to predict the shortest routes between two nodes of a graph by learning the adjacency matrix using the GeoSeq2Seq formalism; our results show that for such a problem the probabilistic representation of the latent embedding supersedes the non-probabilistic embedding by 10-15%.
==== UPDATE AFTER REVIEWER RESPONSE
I apologize to the authors for my late response.
I appreciate the reviewer responses, and they are helpful on a number of
fronts. Still, there are several problematic points.
First, as the authors anticipated, I question whether the geometric encoding
operations can be included in an end-to-end learning setting. I can imagine
several arguments why an end-to-end algorithm may not be preferred, but the
authors do not offer any such arguments.
Second, I am still interested in more discussion of the empirical investigation
into the behavior of the algorithm. For example, "Shortest" and "Successful"
in Table 1 still do not really capture how close "successful but not shortest"
paths are to optimal.
The authors have addressed a number of my concerns, but there
are a few outstanding concerns. Also, other reviewers are much more familiar
with the work than myself. I defer to their judgement after the updates.
==== Original review
In this work, the authors propose an approach to adapt latent representations to account for local geometry in the embedding space. They show modest improvement compared to reasonable baselines.
While I find the idea of incorporating information geometry into embeddings very promising, the current work omits a number of key details that would allow the reader to draw deeper connections between the two (specific comments below). Additionally, the experiments are not particularly insightful.
I believe a substantially revised version of the paper could address most of my concerns; still, I find the current version too preliminary for publication.
=== Major comments / questions
The transformation from context vectors into Fisher vectors is not clear. Presumably, shortest paths in the training data have different lengths, and thus produce different numbers of context vectors. Does the GMM treat all of these independently (regardless of sample)? or is a separate GMM somehow trained for each training sequence? The same question applies to the VLAD-based approach.
In a related vein, it is not clear to what extent this method depends on the sequential nature of the considered networks. In particular, could a similar approach be applied to latent space embeddings from non-sequential models?
It is not clear if the geometric encoding operations are differentiable, or more generally, the entire training algorithm is not clear.
The choice to limit the road network graph feels quite arbitrary. Why was this done?
Deep models are known to be sensitive to the choice of hyperparameters. How were these chosen? was a validation set used in addtion to the training and testing sets?
The target for training is very unclear. Throughout Sections 1 and 2, the aim of the paper appears to be to learn shortest paths; however, Section 3 states that the “network is capable of learning the adjacency matrix”, and the caption for Figure 2 suggests that “[t]he adjacency matrix is iteratively learnt (sic)....” However, calculating such training error for back-propagation/optimization would seem to rely on *already knowing* the adjacency matrix.
The performed experiments are minimal and offer very little insight into what is learned. For example, does the model predict “short” shortest paths better than longer ones? what do the “valid but not optimal” paths look like? are they close to optimal? what do the invalid paths look like? does it seem to learn parts of the road network better than others? sparse parts of the network? dense parts?
=== Minor comments / questions
The term “context vector” is not explicitly defined or described. Based on the second paragraph in the “Fisher encoding” section, I assume these are the latent states for each element in the shortest path sequences.
Is the graph directed? weighted? by Euclidean distance? (Roads are not necessarily straight, so the Euclidean distance from intersection to intersection may not accurately reflect the distance in some cases.)
Are the nodes sampled uniformly at random for creating the training data?
Is the choice to use a diagonal covariance matrix (as opposed to some more flexible one) a computational choice? or does the theory justify this choice?
Roughly, what are the computational resources required for training?
The discussion should explain “condition number” in more detail.
Do the “more precise” results for the Fisher encoding somehow rely on an infinite mixture? or, how much does using only a single component in the GMM affect the results?
It is not clear what “features” and “dictionary elements” are in the context of VLAD.
What value of k was used for K-means clustering for VLAD?
It is not possible to assess the statistical significance of the presented experimental results. More datasets (or different parts of the road network) or cross-validation should be used to provide an indication of the variance of each method.
=== Typos, etc.
The paper includes a number of runon sentences and other small grammatical mistakes. I have included some below.
The first paragraph in Section 2.2 in particular needs to be edited.
The references are inconsistently and improperly (e.g., “Turing” should be capitalized) formatted.
It seems like that $q_{ik} \in \{0,1\}$ for the hard assignments in clustering.