  Contrastive learning has been successfully used for retrieval of semantically
aligned sentences, but it often requires large batch sizes or careful
engineering to work well. In this paper, we instead propose a generative model
for learning multilingual text embeddings which can be used to retrieve or
score sentence pairs. Our model operates on parallel data in $N$ languages and,
through an approximation we introduce, efficiently encourages source separation
in this multilingual setting, separating semantic information that is shared
between translations from stylistic or language-specific variation. We show
careful large-scale comparisons between contrastive and generation-based
approaches for learning multilingual text embeddings, a comparison that has not
been done to the best of our knowledge despite the popularity of these
approaches. We evaluate this method on a suite of tasks including semantic
similarity, bitext mining, and cross-lingual question retrieval -- the last of
which we introduce in this paper. Overall, our Variational Multilingual
Source-Separation Transformer (VMSST) model outperforms both a strong
contrastive and generative baseline on these tasks.

Computation and Language;Machine Learning