Answers.Body: <p>The LSTM seq2seq model typically used for language translation actually can output multiple variations, and does in many implementations.</p>
  
  <p>The decoder stage outputs confidence/probabilities for each word, and it is possible to run the decoder multiple times, taking different samples on each run.</p>
  
  <p>Not only is it possible, but this is actually done by translation engines. A technique called <a href="https://machinelearningmastery.com/beam-search-decoder-natural-language-processing/" rel="nofollow noreferrer">BEAM search</a> is used to manage multiple translations in progress and pick an eventual one to display, based on whether the language model predicts an overall high or low probability for a sentence (this allows for sentences that start with an unusual initial word choice but that are a better choice in context of what is being translated). It would be a minor change to existing algorithms to display a selection of these instead.</p>
  
  <p>Technically you don't need to sample randomly, and can just take the top N probabilities on each step, and recalculate each time on combined probabilities, culling partial sentences that have low probability in favour of continuing more of the high probability sentences. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-11-15T21:57:22.670Z
Answers.Text: What was meant by "taking different samples"? Where is this source of variability? Decoder receives the same input and the same context (from the encoder)..
Answers.UserDisplayName: TomR
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-11-15T21:58:41.837Z
Answers.Text: @TomR: The decoder does not output absolute word choices, it outputs word probabilities. Instead of taking the max probability as the only choice, sample according to the probabilities, using a random number generator. In addition the decoder accepts the previous word and state that it output as input for the next item in the sequence, so as soon as the sentence is started with a different word, this sets the translation on a (possibly very) different trajectory.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-11-15T22:03:08.367Z
Answers.Text: @TomR: Technically you also don't need to sample, just take the top N probabilities on each step, and recalculate each time on combined probabilities, culling partial sentences that have low probability in favour of continuing more of the high probability sentences. That works just fine for BEAM search
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-11-15T22:06:33.127Z
Answers.Text: Many thanks. But is there some published work about such approach or is it folklore shared only among experts (I am novice in this field)? Maybe there is some good keywords. Google gives lot of answers for "multiple output neural network", but not for NMT.
Answers.UserDisplayName: TomR
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-11-15T22:08:28.903Z
Answers.Text: @TomR: I'm pretty sure there are published works. However, I got my understanding of this from Andrew Ng's course Sequence Models. This is a relevant lecture https://www.coursera.org/learn/nlp-sequence-models/lecture/v2pRn/picking-the-most-likely-sentence
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-29T13:59:25.913Z
Answers.Text: I know I am very late to this party, but the ideal search terms are "beam search nmt". One relevant published work is https://www.aclweb.org/anthology/W17-3207.pdf - it also includes references to the works that introduced beam search for sequence models in the first place.
Answers.UserDisplayName: Mathias Müller
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-11-15T21:52:34.753Z
Answers.LastActivityDate: 2018-11-15T22:06:03.700Z
Answers.LastEditDate: 2018-11-15T22:06:03.700Z
Answers.UserDisplayName: Neil Slater
Body: <p>Is there neural machine translation methods, that for one input sentence outputs multiple alternative output sentences in that target language. It is quite possible, that sentence in source language have multiple meanings and it is not desirable that neural network discards some of the meanings if there is no context for disambiguation provided. How multiple outputs can be acommodated into encode-decoder architecture, or different architecture is required?</p>
  
  <p>I am aware of only one work <a href="https://arxiv.org/abs/1805.10844" rel="nofollow noreferrer">https://arxiv.org/abs/1805.10844</a> (and one referen herein) but I am still digesting whether their network outputs multiple sentences or whether it just acommodates variations during training phase.</p>
  
Tags: <neural-networks><computational-linguistics>
Title: Neural machine translation, that outputs multiple alternative, ambiguous translations?
UserDisplayName: TomR
ThreadText: # Neural machine translation, that outputs multiple alternative, ambiguous translations?
  
  (Asked by: username_0 on 2018-11-15)
  
  Is there neural machine translation methods, that for one input sentence outputs multiple alternative output sentences in that target language. It is quite possible, that sentence in source language have multiple meanings and it is not desirable that neural network discards some of the meanings if there is no context for disambiguation provided. How multiple outputs can be acommodated into encode-decoder architecture, or different architecture is required?
  
  I am aware of only one work <https://arxiv.org/abs/1805.10844> (and one referen herein) but I am still digesting whether their network outputs multiple sentences or whether it just acommodates variations during training phase.
  
  **Tags:** neural-networks, computational-linguistics
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-11-15. Score: 1
  
  The LSTM seq2seq model typically used for language translation actually can output multiple variations, and does in many implementations.
  
  The decoder stage outputs confidence/probabilities for each word, and it is possible to run the decoder multiple times, taking different samples on each run.
  
  Not only is it possible, but this is actually done by translation engines. A technique called [BEAM search](https://machinelearningmastery.com/beam-search-decoder-natural-language-processing/) is used to manage multiple translations in progress and pick an eventual one to display, based on whether the language model predicts an overall high or low probability for a sentence (this allows for sentences that start with an unusual initial word choice but that are a better choice in context of what is being translated). It would be a minor change to existing algorithms to display a selection of these instead.
  
  Technically you don't need to sample randomly, and can just take the top N probabilities on each step, and recalculate each time on combined probabilities, culling partial sentences that have low probability in favour of continuing more of the high probability sentences.
  
  > **Comments:**
  > **username_0**: What was meant by "taking different samples"? Where is this source of variability? Decoder receives the same input and the same context (from the encoder)..  
  > **username_1**: @username_0: The decoder does not output absolute word choices, it outputs word probabilities. Instead of taking the max probability as the only choice, sample according to the probabilities, using a random number generator. In addition the decoder accepts the previous word and state that it output as input for the next item in the sequence, so as soon as the sentence is started with a different word, this sets the translation on a (possibly very) different trajectory.  
  > **username_1**: @username_0: Technically you also don't need to sample, just take the top N probabilities on each step, and recalculate each time on combined probabilities, culling partial sentences that have low probability in favour of continuing more of the high probability sentences. That works just fine for BEAM search  
  > **username_0**: Many thanks. But is there some published work about such approach or is it folklore shared only among experts (I am novice in this field)? Maybe there is some good keywords. Google gives lot of answers for "multiple output neural network", but not for NMT.  
  > **username_1**: @username_0: I'm pretty sure there are published works. However, I got my understanding of this from Andrew Ng's course Sequence Models. This is a relevant lecture https://www.coursera.org/learn/nlp-sequence-models/lecture/v2pRn/picking-the-most-likely-sentence  
  > **username_2**: I know I am very late to this party, but the ideal search terms are "beam search nmt". One relevant published work is https://www.aclweb.org/anthology/W17-3207.pdf - it also includes references to the works that introduced beam search for sequence models in the first place.  
  
  ---
  
  