  Models of acoustic word embeddings (AWEs) learn to map variable-length spoken
word segments onto fixed-dimensionality vector representations such that
different acoustic exemplars of the same word are projected nearby in the
embedding space. In addition to their speech technology applications, AWE
models have been shown to predict human performance on a variety of auditory
lexical processing tasks. Current AWE models are based on neural networks and
trained in a bottom-up approach that integrates acoustic cues to build up a
word representation given an acoustic or symbolic supervision signal.
Therefore, these models do not leverage or capture high-level lexical knowledge
during the learning process. In this paper, we propose a multi-task learning
model that incorporates top-down lexical knowledge into the training procedure
of AWEs. Our model learns a mapping between the acoustic input and a lexical
representation that encodes high-level information such as word semantics in
addition to bottom-up form-based supervision. We experiment with three
languages and demonstrate that incorporating lexical knowledge improves the
embedding space discriminability and encourages the model to better separate
lexical categories.

Computation and Language;Audio and Speech Processing