Answers.Body: <p><strong>To answer my own question - it was because of 2 things:</strong> </p>
  
  <ul>
  <li><p><strong>Too small number of batches</strong> - Model just started to gain statistical knowledge about language dialogs. I needed to train it longer.</p></li>
  <li><p><strong>I masked sequence too early (wrongly removed the &lt; END > tag)</strong> - Because on each sentence the last world is just &lt; END > tag - I removed it on all training examples, which prevented Model from learn what does it mean "<em>the end of the sentence</em>".</p></li>
  </ul>
  
  <p>The last condition probably caused that strange pattern even further, because if model doesn't know what word to put in (and because of the lack of the &lt; END > tag) it must fill each sentence till the end of max_sequence_len. </p>
  
  <p>So the Model inputed in <em>loop-like-manner</em> - one of the most common word (where there was no signal from target sentence, because it simply ended).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-14T12:27:44.073Z
Answers.LastActivityDate: 2018-08-14T12:27:44.073Z
Answers.UserDisplayName: Ziemo
Body: <p>I'm training Seq2Seq model on OpenSubtitles dialogs - <a href="http://www.cs.cornell.edu/%7Ecristian/Cornell_Movie-Dialogs_Corpus.html" rel="nofollow noreferrer">Cornell-Movie-Dialogs-Corpus</a>.</p>
  <p>My work based on the following papers (but currently I'm not implemented Attention yet):</p>
  <ul>
  <li><a href="https://arxiv.org/abs/1409.3215" rel="nofollow noreferrer"><em>Sequence to Sequence Learning with Neural Networks, Sutskever et al. 2014</em></a></li>
  <li><a href="https://arxiv.org/abs/1506.05869" rel="nofollow noreferrer"><em>A Neural Conversational Model, Vinyals, Le, 2015</em></a></li>
  </ul>
  <p>The <code>loss</code> I received is quite high and sucked in variation <code>~6.4</code> after 3 epoches. The model predicts the most common words with some times other not significant words (but 99.99% is just 'you'):</p>
  <ul>
  <li>I’ve experimented with 128 - 2048 hidden units and with 1 or 2 or 3 LSTM layers per <code>encoder</code> and <code>decoder</code>. The outcomes are more or less the same.</li>
  </ul>
  <blockquote>
  <p>SEQ1: yeah man it means love respect community and the dollars too the package the unk end</p>
  <p>SEQ2: but how did you get unk 82 end</p>
  <p>PREDICTION: promoting 16th dashboard be of the the the you you you you you you you you you you you you you you you you you you you you you you you you</p>
  </blockquote>
  <p>I'm using here <code>greedy</code> prediction, meaning - after I receive <code>logit</code> I do <code>argmax(..)</code> on all its value for first-3 mini-batch-elements (here I present only first element). For convenient - <code>SEQ1</code> and <code>SEQ2</code> are also printed - to know the actual dialog which was presented to the model.</p>
  <p>The pseudo-code of my architecture looks like this (I'm using Tensorflow 1.5):</p>
  <pre><code>seq1 = tf.placeholder(...)
  seq2 = tf.placeholder(...)
  
  embeddings = tf.Variable(tf.random_uniform([vocab_size, 100],-1,1))
  
  seq1_emb = tf.nn.embedding_lookup(embeddings, seq1)
  seq2_emb = tf.nn.embedding_lookup(embeddings, seq1)
  
  encoder_out, state1 = tf.nn.static_rnn(BasicLSTMCell(), seq1_emb)
  decoder_out, state2 = tf.nn.static_rnn(BasicLSTMCell(), seq2_emb,
                                                          initial_state=state_1)
  logit = Dense(decoder_out, use_bias=False)
  
  crossent = tf.nn.saparse_softmax_cross_entropy_with_logits(logits=logit, 
                                                           labels=target)
  crossent = mask_padded_zeros(crossent)
  loss = tf.reduce_sum(crossent) / number_of_words_in_batch
  
  train = tf.train.AdamOptimizer(learning_rate=0.00002).minimize(loss) 
  </code></pre>
  <p>I'm also wonder if I pass well <code>state1</code> to <code>decoder</code>, which in general looks like this:</p>
  <pre><code># reshape in pseudocode: state1 = state[1:]
  new_state1 = []
  for lstm in state1:
      new_lstm = []
      for gate in lstm:
          new_lstm.append(gate[1:])
      new_state1.append(tuple(new_lstm))
  state1 = tuple(new_state1)
  </code></pre>
  <ul>
  <li>Should I use some projection layer between states of <code>encoder</code> and <code>decoder</code> ?</li>
  </ul>
  <p>So if <code>seq1</code> has 32 words, <code>seq2</code> has 31 (since we will not predict nothing after the last word, which is the tag <code>&lt;END&gt;</code>).</p>
  
Tags: <neural-networks><natural-language-processing><tensorflow><recurrent-neural-networks><long-short-term-memory>
Title: Seq2Seq dialogs predicts only most common words like `you` after couple of epoches
UserDisplayName: Ziemo
ThreadText: # Seq2Seq dialogs predicts only most common words like `you` after couple of epoches
  
  (Asked by: username_0 on 2018-03-06)
  
  I'm training Seq2Seq model on OpenSubtitles dialogs - Cornell-Movie-Dialogs-Corpus.
  My work based on the following papers (but currently I'm not implemented Attention yet):
  
  Sequence to Sequence Learning with Neural Networks, Sutskever et al. 2014
  A Neural Conversational Model, Vinyals, Le, 2015
  
  The loss I received is quite high and sucked in variation ~6.4 after 3 epoches. The model predicts the most common words with some times other not significant words (but 99.99% is just 'you'):
  
  I’ve experimented with 128 - 2048 hidden units and with 1 or 2 or 3 LSTM layers per encoder and decoder. The outcomes are more or less the same.
  
  
  SEQ1: yeah man it means love respect community and the dollars too the package the unk end
  SEQ2: but how did you get unk 82 end
  PREDICTION: promoting 16th dashboard be of the the the you you you you you you you you you you you you you you you you you you you you you you you you
  
  I'm using here greedy prediction, meaning - after I receive logit I do argmax(..) on all its value for first-3 mini-batch-elements (here I present only first element). For convenient - SEQ1 and SEQ2 are also printed - to know the actual dialog which was presented to the model.
  The pseudo-code of my architecture looks like this (I'm using Tensorflow 1.5):
  seq1 = tf.placeholder(...)
  seq2 = tf.placeholder(...)
  
  embeddings = tf.Variable(tf.random_uniform([vocab_size, 100],-1,1))
  
  seq1_emb = tf.nn.embedding_lookup(embeddings, seq1)
  seq2_emb = tf.nn.embedding_lookup(embeddings, seq1)
  
  encoder_out, state1 = tf.nn.static_rnn(BasicLSTMCell(), seq1_emb)
  decoder_out, state2 = tf.nn.static_rnn(BasicLSTMCell(), seq2_emb,
                                                          initial_state=state_1)
  logit = Dense(decoder_out, use_bias=False)
  
  crossent = tf.nn.saparse_softmax_cross_entropy_with_logits(logits=logit, 
                                                           labels=target)
  crossent = mask_padded_zeros(crossent)
  loss = tf.reduce_sum(crossent) / number_of_words_in_batch
  
  train = tf.train.AdamOptimizer(learning_rate=0.00002).minimize(loss) 
  
  I'm also wonder if I pass well state1 to decoder, which in general looks like this:
  # reshape in pseudocode: state1 = state[1:]
  new_state1 = []
  for lstm in state1:
      new_lstm = []
      for gate in lstm:
          new_lstm.append(gate[1:])
      new_state1.append(tuple(new_lstm))
  state1 = tuple(new_state1)
  
  
  Should I use some projection layer between states of encoder and decoder ?
  
  So if seq1 has 32 words, seq2 has 31 (since we will not predict nothing after the last word, which is the tag ).
  
  **Tags:** neural-networks, natural-language-processing, tensorflow, recurrent-neural-networks, long-short-term-memory
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2018-08-14. Score: 0 | Accepted answer
  
  **To answer my own question - it was because of 2 things:**
  
  * **Too small number of batches** - Model just started to gain statistical knowledge about language dialogs. I needed to train it longer.
  * **I masked sequence too early (wrongly removed the < END > tag)** - Because on each sentence the last world is just < END > tag - I removed it on all training examples, which prevented Model from learn what does it mean "*the end of the sentence*".
  
  The last condition probably caused that strange pattern even further, because if model doesn't know what word to put in (and because of the lack of the < END > tag) it must fill each sentence till the end of max\_sequence\_len.
  
  So the Model inputed in *loop-like-manner* - one of the most common word (where there was no signal from target sentence, because it simply ended).
  
  
  ---
  
  