Unsupervised Cross-Modal Alignment of Speech and Text Embedding Spaces
Recent research has shown that word embedding spaces learned from text corpora of different languages can be aligned without any parallel data supervision. Inspired by the success in unsupervised cross-lingual word embeddings, in this paper we target learning a cross-modal alignment between the embedding spaces of speech and text learned from corpora of their respective modalities in an unsupervised fashion. The proposed framework learns the individual speech and text embedding spaces, and attempts to align the two spaces via adversarial training, followed by a refinement procedure. We show how our framework could be used to perform spoken word classification and translation, and the experimental results on these two tasks demonstrate that the performance of our unsupervised alignment approach is comparable to its supervised counterpart. Our framework is especially useful for developing automatic speech recognition (ASR) and speech-to-text translation systems for low-or zero-resource languages, which have little parallel audio-text data for training modern supervised ASR and speech-to-text translation models, but account for the majority of the languages spoken across the world.
This paper investigates unsupervised alignment between text and speech using embedding for both modalities.  Word and speech embedding are first generated, the speech embedding are computed using an unsupervised alignment procedure and averaged. The alignment model is then trained based on adversarial training. The proposed approach is then evaluated on two tasks: spoken word detection and spoken word translation.   Quality: The results of the spoken word detection study are not very appealing: 89% accuracy for the supervised baseline versus 23.7% for the unsupervised proposed approach is rather poor. But the synonym retrieval results are very interesting. Showing that the speech embedding seems to capture semantics information instead of phonetic is significant. The spoken word translation experiment is also interesting. For the spoken word detection task, it's not very clear: what is the input exactly ? is it a full utterance containing several words including the target one, or was the utterance aligned and cropped to contain only the uttered word ? I also think the authors should select a more established task instead of the one they proposed, spoken term detection seems relevant [1].   Clarity: the paper is well-written and easy to read.  Originality: This work is a novel combination of established techniques, such as word embedding and adversarial training. The unsupervised method to compute speech embedding is novel to my knowledge.  Significance: Unsupervised sequence classification, such as ASR and MT, is definitely the next huge milestone in the field, and this paper is significant step towards that long-term objective.   Overall, despite the not-so-great results, I think the paper should be accepted as it presents a good lead toward unsupervised sequence classification.  [1] J. G. Fiscus, J. Ajot, J. Garofolo, and G. Doddingtion, “Results of the 2006 spoken term detection evaluation,”  in Proc. SIGIR, vol. 7. Citeseer, 2007, pp. 51–57.  UPDATE: The A vs A* result is very interesting and is a very good result to demonstrate the potential of the approach. I suggest that this result should be better highlighted in the final version paper