Learning semantic similarity in a continuous space
We address the problem of learning semantic representation of questions to measure similarity between pairs as a continuous distance metric. Our work naturally extends Word Mover's Distance (WMD) [1] by representing text documents as normal distributions instead of bags of embedded words. Our learned metric measures the dissimilarity between two questions as the minimum amount of distance the intent (hidden representation) of one question needs to "travel" to match the intent of another question. We first learn to repeat, reformulate questions to infer intents as normal distributions with a deep generative model [2] (variational auto encoder). Semantic similarity between pairs is then learned discriminatively as an optimal transport distance metric (Wasserstein 2) with our novel variational siamese framework. Among known models that can read sentences individually, our proposed framework achieves competitive results on Quora duplicate questions dataset. Our work sheds light on how deep generative models can approximate distributions (semantic representations) to effectively measure semantic similarity with meaningful distance metrics from Information Theory.
The paper addresses the problem of learning the representation of questions/sentences as an input unit. Like word2gauss, the embedding is into distributions (as opposed to a single point of a vector space) and in particular into Gaussian distributions. Embedding into a distribution enables the use of a transport distance, such as W2 for computing discrepancy between distributions which has advantages over alternatives such as KL divergence but is expensive to compute. Here the specific constraints on the distributions made the computation of the transport distance efficient. The application considered is question de-duplication. The authors solve the problem in two separate stages. The first stage is the actual representation learning stage, which embeds questions into the Gaussian distribution. The second stage is to learn from labeled data to decide if the two pre-trained representations correspond to equivalent questions or not. For the first stage a generative (LSTM based) VAE model is proposed to convert a question into itself or a similar question. For the second stage a Siamese structure is proposed that is based on W2 distances. The model is evaluated on Quora question deduplication dataset and performed strongly.   I liked this paper: there is a creative component in the overall proposed solution and the architecture and the pieces fit together well. The embedding of a whole sentence to a distribution seems very natural, W2 seems like a good way to compute distances, etc. I also liked the idea of "similar-encoder", which could also be useful as opposed to "auto-encoder", as an augmentation method. The two stage development of the solution raises my curiosity though. The first stage, i.e. text generation, is a difficult task. Eliminating this stage completely and performing an end-to-end training would have been an alternative way to go. The pre-training from the first stage could still be used for initialization, but the question representations could have been fine-tuned for the actual task. I was hoping to see more discussions/results on that alternative. Moreover, it is surprising that no generation result is reported.     The paper is well organized and readable. But it needs proofreading and improvement in presentation in places. Mathematical notations should be more carefully used. - The notations for vectors and scalars seem to be mixed up.  See for example Section 3.2. There seem to be a dimension mismatch in Equations (3) and (4). The notation for matrix square root is unclear.  - Section 4.2 is not clear. What is "element-wise" Wasserstein? Is it not that Wasserstein distance returns a number (scalar) for a pair of vectors? Is the dimension of the input to the next module h+1? (One for the W norm and h for the Hadamard product?) - Not enough details is given about the W2 computation for Gaussians. In particular, the first paragraph of page 6 (in Section 4.1) lacks enough details and is cryptic. Needs expansion. - Minor:  "Similarly to","Sigm", "sentences representation", "euclidean", "gaussian", "figure 3", "[3][4][5][6] and [7]", "87,5".  The approach proposed is novel to my knowledge. The work is a good continuation of the line of research that focuses on embedding into distributions. It is also a good application of W2 distance between distributions. The generation part of the proposal ( "Repeat-Reformulate") reminds me of "Sequence to better sequence" in [1]. Citation might be a good idea.  The significance of the method is enough to be accepted I think. Learning representation for a question/sentence is an important and unsolved problem. Embedding to the right mathematical object is an important question and explorations and insights on that seems useful. Coming up with a robust distance between distributions that is easy to compute is also beneficial. Finally, the application considered has received attention lately in ML and NLP community.  Overall, considering strengths and weaknesses of the paper, I vote for the paper to be accepted.  ---------------------------- [1] Mueller, J., Gifford, D., Jaakkola, T.  "Sequence to better sequence: continuous revision of combinatorial structures",  in ICML 2017. ---------------------------- Thanks for the clarification on the "element-wise" Wasserstein operation that is mentioned. It makes more sense now. That should definitely be clarified in the write up. (Mathematically defining the ith element of the operation by using indices would be helpful to clarify the definition.)