Semantic sentence embedding models encode natural language sentences into vectors, such that closeness in embedding space indicates closeness in the semantics between the sentences. Bilingual data offers a useful signal for learning such embeddings: properties shared by both sentences in a translation pair are likely semantic, while divergent properties are likely stylistic or language-specific. We propose a deep latent variable model that attempts to perform source separation on parallel sentences, isolating what they have in common in a latent semantic vector, and explaining what is left over with language-specific latent vectors. Our proposed approach differs from past work on semantic sentence encoding in two ways. First, by using a variational probabilistic framework, we introduce priors that encourage source separation, and can use our model's posterior to predict sentence embeddings for monolingual data at test time. Second, we use highcapacity transformers as both data generating distributions and inference networks -contrasting with most past work on sentence embeddings. In experiments, our approach substantially outperforms the state-of-the-art on a standard suite of unsupervised semantic similarity evaluations. Further, we demonstrate that our approach yields the largest gains on more difficult subsets of these evaluations where simple word overlap is not a good indicator of similarity.
This paper addresses the problem of constructing a sentence embedding using a generative transformer model which encodes semantic aspects and language-specific aspect separately. They use transformers to encode and decode sentence embedding, and the objective reconstructs input with a latent variables (language variables for each language and semantic language).  These latent variables are sampled from multivariate Gaussian prior, and the learning uses evidence lower bound (ELBO) for variational approximation of the joint distribution of latent variables and input. 
The method is evaluated on two tasks: sentence similarity task and machine translation evaluation metric tasks. Both tasks evaluates how similar are two sequences, and the metric is correlation score with score’s from human judge. The model shows promising results on the first task, but weaker results on the second task, especially when compared against pretty naively built sentence embedding from BERT model. I’m not expert in sentence embedding literature, so a bit tricky to evaluate, but baselines seem strong and experimental results on semantic textual similarity task.
In terms of evaluation, I appreciated how they defined harder subset of the evaluation dataset and showed a larger improvements on those portions of the dataset. The The paper also includes analysis on what is captured by their language-specific latent vector and semantic latent vector. While I’m not totally convinced this distinction between language-specific characteristics and semantics of the sentence, it makes it easier to understand what’s going on in the model. 
One of my question is, why not test this method in more popular benchmark such as MNLI or other classification tasks? MNLI evaluates how each sentence pair relates to one another, thus would be a good benchmark for sentence embeddings as well. Having to encode all the information about a sentence into a single vector will make these sentence embedding model weaker than other models which can do cross sentence attentions and etc, but I think that’s the genuine limitation of sentence embedding research and has to be clarified as such. I recommend discussing and clarifying these points. 
I’m a bit unclear how these sentence embeddings are translated into a score that decides the degree to which sentences have the same meaning. Is it just cosine similarity of two sentence embedding vectors?
While the purpose of these references is to generate sentences instead of building a sentence embedding, the method is related and comparison and discussion would be worthwhile. 
Generating Sentences from a Continuous Space
Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew M. Dai, Rafal Jozefowicz, Samy Bengio
https://arxiv.org/abs/1511.06349
Toward Controlled Generation of Text
Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, Eric P. Xing
https://arxiv.org/abs/1703.00955
Comments & Questions:
- Methods using a large amount of unsupervised monolingual data shows very strong performance in a panoply of NLP tasks these days. If I understand correctly, this model is constrained by the amount of bitext — some analysis on this would be interesting. 
- Figure 1 mentions about “Section 3, 4” but I don’t think they are correct references? 
- BERT baseline seemed not to allow fine-tuning of the LM parameters. I think this makes the baseline significantly weaker? 
- It seems odd that only English semantic encoder is used to downstream application.
- Does table 3 covers all the data? What proportion of the data is covered by each row?
- Given the similarity of English and French, I’m not sure how “language-specific” such latent vectors are. It would be much more interesting analysis if it studies distant language pairs.