What is reordering in the context of the paper?
Re-arranging translated words so that they are in the correct order in the target language
Unsupervised learning of cross-lingual word embedding offers elegant matching of words across languages, but has fundamental limitations in translating sentences. In this paper, we propose simple yet effective methods to improve word-by-word translation of cross-lingual embeddings, using only monolingual corpora but without any back-translation. We integrate a language model for context-aware search, and use a novel denoising autoencoder to handle reordering. Our system surpasses state-of-the-art unsupervised neural translation systems without costly iterative training. We also analyze the effect of vocabulary size and denoising type on the translation performance, which provides better understanding of learning the cross-lingual word embedding and its usage in translation.
Building a machine translation (MT) system requires lots of bilingual data. Neural MT models BIBREF0 , which become the current standard, are even more difficult to train without huge bilingual supervision BIBREF1 . However, bilingual resources are still limited to some of the selected language pairs—mostly from or to English. A workaround for zero-resource language pairs is translating via an intermediate (pivot) language. To do so, we need to collect parallel data and train MT models for source-to-pivot and pivot-to-target individually; it takes a double effort and the decoding is twice as slow. Unsupervised learning is another alternative, where we can train an MT system with only monolingual corpora. Decipherment methods BIBREF2 , BIBREF3 are the first work in this direction, but they often suffer from a huge latent hypothesis space BIBREF4 . Recent work by unmt-artetxe and unmt-facebook train sequence-to-sequence MT models of both translation directions together in an unsupervised way. They do back-translation BIBREF5 back and forth for every iteration or batch, which needs an immensely long time and careful tuning of hyperparameters for massive monolingual data. Here we suggest rather simple methods to build an unsupervised MT system quickly, based on word translation using cross-lingual word embeddings. The contributions of this paper are: The proposed models can be efficiently trained with off-the-shelf softwares with little or no changes in the implementation, using only monolingual data. The provided analyses help for better learning of cross-lingual word embeddings for translation purpose. Altogether, our unsupervised MT system outperforms the sequence-to-sequence neural models even without training signals from the opposite translation direction, i.e. via back-translation.