Body: <p>I am trying to build an LSTM model to generate Shakspeare-like poems. I have training set <span class="math-container">$\{s_1,s_2, \dots,s_m\}$</span>, which are sentences of Shakespeare poems, and each sentence contains words <span class="math-container">$\{w_1,w_2, \dots,w_n\}$</span>.</p>
  
  <p>To my understanding, each sentence <span class="math-container">$s_i$</span>, for <span class="math-container">$i=1, \dots,m$</span> is a random sequence containing the words <span class="math-container">$w_j$</span>, for <span class="math-container">$j=1, \dots,n$</span>. The LSTM model is estimated by applying the maximum likelihood (MLE) method, which will use cross-entropy loss for optimization. The use of MLE requires that the samples in the random sequence be independent and identically distributed (i.i.d), however, the word sequence <span class="math-container">$w_j$</span> is not i.i.d (since it is non-Markov). Therefore, I am suspicious about using cross-entropy loss for training an LSTM for the NLP task (which seems to be the common practice). </p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-03-08T02:03:35.007Z
Comments.Text: The negative log-likelihood should indeed be equivalent to the cross-entropy, provided that the independence assumption is made. In fact, I have derived their equivalence in my notes by making this assumption. So, if you use the cross-entropy, you're implicitly making the independence assumption.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-03-09T03:18:51.667Z
Comments.Text: Thanks for the edit and reply. I agree that negative log-likelihood is equivalent to cross-entropy when independence assumption is made. But for a NLP task, where the distribution for the next word is clearly not independent and identical to that of previous words, I am very suspicious on the adoption of cross-entropy loss. I wonder if there are any reasoning for using it or we can use any other loss for such task.
Comments.UserDisplayName: Leey
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-03-09T03:26:03.310Z
Comments.Text: Well, while training your model, you will maybe provide the sentences (or words) in random order, this should have a "decorrelation effect". However, I don't know if this is sufficient. I would need to review my notes about CE and NLL and where exactly do we assume independence. Also, in some cases (e.g. naive Bayes), the independence assumption is made (only for simplicity and because it sometimes works), while it often doesn't hold.
Comments.UserDisplayName: nbro
Tags: <long-short-term-memory><cross-entropy><maximum-likelihood>
Title: Can the cross-entropy loss be used for a NLP task with LSTM?
UserDisplayName: Leey
ThreadText: # Can the cross-entropy loss be used for a NLP task with LSTM?
  
  (Asked by: username_0 on 2020-03-08)
  
  I am trying to build an LSTM model to generate Shakspeare-like poems. I have training set $\{s\_1,s\_2, \dots,s\_m\}$, which are sentences of Shakespeare poems, and each sentence contains words $\{w\_1,w\_2, \dots,w\_n\}$.
  
  To my understanding, each sentence $s\_i$, for $i=1, \dots,m$ is a random sequence containing the words $w\_j$, for $j=1, \dots,n$. The LSTM model is estimated by applying the maximum likelihood (MLE) method, which will use cross-entropy loss for optimization. The use of MLE requires that the samples in the random sequence be independent and identically distributed (i.i.d), however, the word sequence $w\_j$ is not i.i.d (since it is non-Markov). Therefore, I am suspicious about using cross-entropy loss for training an LSTM for the NLP task (which seems to be the common practice).
  
  **Tags:** long-short-term-memory, cross-entropy, maximum-likelihood
  > **Comments:**
  > **username_1**: The negative log-likelihood should indeed be equivalent to the cross-entropy, provided that the independence assumption is made. In fact, I have derived their equivalence in my notes by making this assumption. So, if you use the cross-entropy, you're implicitly making the independence assumption.  
  > **username_0**: Thanks for the edit and reply. I agree that negative log-likelihood is equivalent to cross-entropy when independence assumption is made. But for a NLP task, where the distribution for the next word is clearly not independent and identical to that of previous words, I am very suspicious on the adoption of cross-entropy loss. I wonder if there are any reasoning for using it or we can use any other loss for such task.  
  > **username_1**: Well, while training your model, you will maybe provide the sentences (or words) in random order, this should have a "decorrelation effect". However, I don't know if this is sufficient. I would need to review my notes about CE and NLL and where exactly do we assume independence. Also, in some cases (e.g. naive Bayes), the independence assumption is made (only for simplicity and because it sometimes works), while it often doesn't hold.  
  
  ---
  
  ## 0 answers
  
  