Answers.Body: <p>The trained model predicts the probability of a given sequence of tokens. Whatever NLP task you are doing, you usually want to get a high-probability sample from that probability distribution. This sampling task could be quite non-trivial.</p>
  <p>What you are seeing is most likely the result of a <a href="https://en.wikipedia.org/wiki/Greedy_algorithm" rel="nofollow noreferrer">greedy</a> sampling - the most probable next word is chosen from the probability distribution. This quite often leads to infinite repletion loops as you experience.</p>
  <p>Simplest way to solve the repetition problem is to actually randomly sample from the probability distribution of the next word - the distribution is usually augmented with so-called <a href="https://cs.stackexchange.com/questions/79241/what-is-temperature-in-lstm-and-neural-networks-generally">temperature</a> parameter.</p>
  <p>Finally, there's <a href="https://en.wikipedia.org/wiki/Beam_search" rel="nofollow noreferrer">beam search</a> - you just perform a tree search for a best N-gram sample in the future. That's the most sophisticated, and the most computationally hard one.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-25T23:08:23.503Z
Answers.LastActivityDate: 2021-04-25T23:08:23.503Z
Answers.UserDisplayName: Kostya
Body: <p>My framework is an encoder-decoder (LSTM-to-LSTM) model, similar to <a href="https://colab.research.google.com/github/bentrevett/pytorch-seq2seq/blob/master/1%20-%20Sequence%20to%20Sequence%20Learning%20with%20Neural%20Networks.ipynb" rel="nofollow noreferrer">this post</a>. The model basically reads a sentence and generate another sentence. But, the thing is, after a few epochs training, the model cannot produce any meaningful output, instead it keeps generating repeating words.</p>
  <p>The framework can translate Franch-Enlgish very effectively, but for my problem it generates the result like this.</p>
  <p>Can you explain why it produces such result, Thank you</p>
  <ul>
  <li>Here is my printed output:</li>
  </ul>
  <p><a href="https://i.sstatic.net/sPU7o.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/sPU7o.png" alt="The screenshot from the output" /></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-26T00:59:21.817Z
Comments.Text: Hello. Although not strictly necessary in this case, rather than providing a screenshot of the output, it would be better if you copy and paste the output. You should also provide more details about your model, how you trained it, etc., because saying that your model is similar to another may not be sufficient to fully answer your question.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-26T01:02:36.037Z
Comments.Text: Yes your are right, I would update it after I finish the experiment.
Comments.UserDisplayName: Cheleeger Ken
Tags: <recurrent-neural-networks><long-short-term-memory><seq2seq><encoder-decoder>
Title: Seq2Seq model produces repeating words
UserDisplayName: Cheleeger Ken
ThreadText: # Seq2Seq model produces repeating words
  
  (Asked by: username_0 on 2021-04-25)
  
  My framework is an encoder-decoder (LSTM-to-LSTM) model, similar to [this post](https://colab.research.google.com/github/bentrevett/pytorch-seq2seq/blob/master/1%20-%20Sequence%20to%20Sequence%20Learning%20with%20Neural%20Networks.ipynb). The model basically reads a sentence and generate another sentence. But, the thing is, after a few epochs training, the model cannot produce any meaningful output, instead it keeps generating repeating words.
  
  The framework can translate Franch-Enlgish very effectively, but for my problem it generates the result like this.
  
  Can you explain why it produces such result, Thank you
  
  * Here is my printed output:
  
  [![The screenshot from the output](https://i.sstatic.net/sPU7o.png)](https://i.sstatic.net/sPU7o.png)
  
  **Tags:** recurrent-neural-networks, long-short-term-memory, seq2seq, encoder-decoder
  > **Comments:**
  > **username_1**: Hello. Although not strictly necessary in this case, rather than providing a screenshot of the output, it would be better if you copy and paste the output. You should also provide more details about your model, how you trained it, etc., because saying that your model is similar to another may not be sufficient to fully answer your question.  
  > **username_0**: Yes your are right, I would update it after I finish the experiment.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-04-25. Score: 1
  
  The trained model predicts the probability of a given sequence of tokens. Whatever NLP task you are doing, you usually want to get a high-probability sample from that probability distribution. This sampling task could be quite non-trivial.
  
  What you are seeing is most likely the result of a [greedy](https://en.wikipedia.org/wiki/Greedy_algorithm) sampling - the most probable next word is chosen from the probability distribution. This quite often leads to infinite repletion loops as you experience.
  
  Simplest way to solve the repetition problem is to actually randomly sample from the probability distribution of the next word - the distribution is usually augmented with so-called [temperature](https://cs.stackexchange.com/questions/79241/what-is-temperature-in-lstm-and-neural-networks-generally) parameter.
  
  Finally, there's [beam search](https://en.wikipedia.org/wiki/Beam_search) - you just perform a tree search for a best N-gram sample in the future. That's the most sophisticated, and the most computationally hard one.
  
  
  ---
  
  