Answers.Body: <p>The problem is not that RNN flavours such as LSTMs are incapable of keeping track of the "important" parts of the input. They also do not have much trouble recognizing commas in different places.</p>
  
  <p>To prove this point, I recommend reading <a href="http://karpathy.github.io/2015/05/21/rnn-effectiveness/" rel="nofollow noreferrer">Andrej Karpathy's excelllent write-up about the behaviour of individual RNN "neurons"</a>.</p>
  
  <p>Addressing specifically this comment in your question:</p>
  
  <blockquote>
    <p>I suspect another reason why the LSTM is not able to infer the format is that the comma can be placed in different indexes of the sequence, so it could be losing its importance in the hidden state the longer the sequence is (not sure if that makes sense).</p>
  </blockquote>
  
  <p>If commas are relevant to solve the task at hand, LSTMs can learn to remember its position or related information. This information is not necessarily diluted by repeated application of reccurrence with long sequences: networks can learn to propagate and promote crucial information from one recurrent state to the next.</p>
  
  <hr>
  
  <p><strong>Input sequences have arbitrary length, which means that LSTMs need to compress information about seen sequence elements</strong></p>
  
  <p>Rather, the input sequence has an arbitrary length, while the LSTM state vectors have a fixed size. State vectors are the only way for an LSTM to "keep track of important parts". This means that those fixed-size vectors are a bottleneck and there is an information-theoretic upper bound on the amount of information about "important parts" that can be kept by an LSTM.</p>
  
  <p><strong>LSTMs potentially take multiple decisions. For each decision, something else in the input sequence is most important</strong></p>
  
  <p>For tasks such as summarization that you mention in the question, an LSTM makes a series of predictions (predicting the tokens of the summary one token at a time). For each prediction, different things in the input sequence might be important. Put another way, for each decision, another <em>view</em> of the input may be most helpful.</p>
  
  <p>This is a key motivation for using attention networks. Each time an LSTM is making a prediction, an attention network can provide a dynamic, optimally helpful view of the input sequence.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-29T12:52:27.837Z
Answers.LastActivityDate: 2020-01-29T12:52:27.837Z
Answers.UserDisplayName: Mathias Müller
Answers.Body: <p>You can do custom POS Tagging and use it as a multi featured sequence2sequence. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-27T09:35:17.003Z
Answers.Text: I think that might still run into the same issues because the comma and positions end up varying based on name so the position isn't as much of a relevant feature
Answers.UserDisplayName: user8714896
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-27T06:34:00.160Z
Answers.LastActivityDate: 2019-12-27T06:34:00.160Z
Answers.UserDisplayName: Ishaan Sharma
Body: <p>I keep reading about how LSTMs can't remember the "important parts" of a sequence which is why attention-based mechanisms are required. I was trying to use LSTMs to find people's name format. </p>
  
  <p>For example, "Millie Bobby Brown" can be seen as first_name middle_name last_name format, which I'll denote as 0, but then there's "Brown, Millie Bobby" which is last_name, first_name middle_name, which I'll denote as 1. </p>
  
  <p>The LSTM seems to be overfitting to one classification of format. I suspect it's because it's not paying special attention to the comma which is a key feature of what format it could be. I'm trying to understand why an LSTM won't work for a case like this. It makes sense to me because LSTMs are better at identifying sequence to sequence generation and things such as summarization and sentiment analysis usually require attention. I suspect another reason why the LSTM is not able to infer the format is that the comma can be placed in different indexes of the sequence, so it could be losing its importance in the hidden state the longer the sequence is (not sure if that makes sense). Anyone else has any theories? I'm trying to convince my fellow researchers that a pure LSTM won't be sufficient for this problem.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-12-26T05:56:17.957Z
Comments.Text: Can you provide examples of misclassifications.
Comments.UserDisplayName: respectful
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-12-26T21:12:01.710Z
Comments.Text: Have you tried a highly naive algorithm that outputs 1 in there is a comma and 0 otherwise?
Comments.UserDisplayName: respectful
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-12-27T01:05:25.390Z
Comments.Text: This could depend on a lot of things, so if you could provide the architecture you're using, that would help. For example, if you were to try and use regression for this (ie, 1 input node: a=0,b=1,c=2... until ,=26), most likely you will get very bad results. If you are doing this, make the input a 1 hot vector of size 27 for all letters and comma. You'll likely get better results.
Comments.UserDisplayName: Recessive
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-12-27T03:38:29.097Z
Comments.Text: @Recessive the architecture is using an autoencoder (Because I'm also trying to generate infer what is the first, middle and last name using stochastic variational inference). The encoder is made using an LSTM because hopefully it'd keep track of the position of the comma and spaces. Then that hidden state is passed into a standard nn for classification.
Comments.UserDisplayName: user8714896
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-12-27T03:40:08.177Z
Comments.Text: @Recessive I'm very convinced if I embed the inputs better that I could just use a nn for very accurate classification. Like "Millie Bobby Brown" Can be embedded as "10101", 1=word, 0 =space and 2=comma. Because for format we don't actually care about the names itself, just the spaces and comma. So for format inference there's a lot of useless features being stored in the hidden state that are just added noise.
Comments.UserDisplayName: user8714896
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-12-27T05:06:12.343Z
Comments.Text: @Recessive yea you essentially got the architecture down. That's essentially what's happening for encoding, but I think that encoding has a lot of excess information not useful for classification of format. I think convolutions might even perform better if each convolution emphasizes if a subset of the name has a comma or space.
Comments.UserDisplayName: user8714896
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-12-28T01:27:58.363Z
Comments.Text: @user8714896 Try using an LSTM with 16 hidden units, an input size of 3 (a 1 hot vector index 1 = word, index 2 = space and index 3 = comma) and output size of 1. Drop the autoencoder (also you should specify it is variational because they're 2 different things) altogether and just perform outright classification. However, in this case, I think you'll get an accuracy of 90% or higher by just outputting 1 if there's a comma, and 0 otherwise by the looks of it
Comments.UserDisplayName: Recessive
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-12-30T17:07:53.063Z
Comments.Text: @user8714896 what is the question exactly? Is it why LSTM do not work for that case? Because you seem to already know the solution (using attention mechanisms).
Comments.UserDisplayName: Miguel Saraiva
Tags: <neural-networks><long-short-term-memory><attention>
Title: Why can't LSTMs keep track of the "important parts" of a sequence?
UserDisplayName: user8714896
ThreadText: # Why can't LSTMs keep track of the "important parts" of a sequence?
  
  (Asked by: username_0 on 2019-12-21)
  
  I keep reading about how LSTMs can't remember the "important parts" of a sequence which is why attention-based mechanisms are required. I was trying to use LSTMs to find people's name format.
  
  For example, "Millie Bobby Brown" can be seen as first\_name middle\_name last\_name format, which I'll denote as 0, but then there's "Brown, Millie Bobby" which is last\_name, first\_name middle\_name, which I'll denote as 1.
  
  The LSTM seems to be overfitting to one classification of format. I suspect it's because it's not paying special attention to the comma which is a key feature of what format it could be. I'm trying to understand why an LSTM won't work for a case like this. It makes sense to me because LSTMs are better at identifying sequence to sequence generation and things such as summarization and sentiment analysis usually require attention. I suspect another reason why the LSTM is not able to infer the format is that the comma can be placed in different indexes of the sequence, so it could be losing its importance in the hidden state the longer the sequence is (not sure if that makes sense). Anyone else has any theories? I'm trying to convince my fellow researchers that a pure LSTM won't be sufficient for this problem.
  
  **Tags:** neural-networks, long-short-term-memory, attention
  
  > **Comments:**
  > **username_1**: Can you provide examples of misclassifications.  
  > **username_1**: Have you tried a highly naive algorithm that outputs 1 in there is a comma and 0 otherwise?  
  > **username_2**: This could depend on a lot of things, so if you could provide the architecture you're using, that would help. For example, if you were to try and use regression for this (ie, 1 input node: a=0,b=1,c=2... until ,=26), most likely you will get very bad results. If you are doing this, make the input a 1 hot vector of size 27 for all letters and comma. You'll likely get better results.  
  > **username_0**: @username_2 the architecture is using an autoencoder (Because I'm also trying to generate infer what is the first, middle and last name using stochastic variational inference). The encoder is made using an LSTM because hopefully it'd keep track of the position of the comma and spaces. Then that hidden state is passed into a standard nn for classification.  
  > **username_0**: @username_2 I'm very convinced if I embed the inputs better that I could just use a nn for very accurate classification. Like "Millie Bobby Brown" Can be embedded as "10101", 1=word, 0 =space and 2=comma. Because for format we don't actually care about the names itself, just the spaces and comma. So for format inference there's a lot of useless features being stored in the hidden state that are just added noise.  
  > **username_0**: @username_2 yea you essentially got the architecture down. That's essentially what's happening for encoding, but I think that encoding has a lot of excess information not useful for classification of format. I think convolutions might even perform better if each convolution emphasizes if a subset of the name has a comma or space.  
  > **username_2**: @username_0 Try using an LSTM with 16 hidden units, an input size of 3 (a 1 hot vector index 1 = word, index 2 = space and index 3 = comma) and output size of 1. Drop the autoencoder (also you should specify it is variational because they're 2 different things) altogether and just perform outright classification. However, in this case, I think you'll get an accuracy of 90% or higher by just outputting 1 if there's a comma, and 0 otherwise by the looks of it  
  > **username_3**: @username_0 what is the question exactly? Is it why LSTM do not work for that case? Because you seem to already know the solution (using attention mechanisms).  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_4 on 2020-01-29. Score: 1 | Accepted answer
  
  The problem is not that RNN flavours such as LSTMs are incapable of keeping track of the "important" parts of the input. They also do not have much trouble recognizing commas in different places.
  
  To prove this point, I recommend reading [Andrej Karpathy's excelllent write-up about the behaviour of individual RNN "neurons"](http://karpathy.github.io/2015/05/21/rnn-effectiveness/).
  
  Addressing specifically this comment in your question:
  
  > I suspect another reason why the LSTM is not able to infer the format is that the comma can be placed in different indexes of the sequence, so it could be losing its importance in the hidden state the longer the sequence is (not sure if that makes sense).
  
  If commas are relevant to solve the task at hand, LSTMs can learn to remember its position or related information. This information is not necessarily diluted by repeated application of reccurrence with long sequences: networks can learn to propagate and promote crucial information from one recurrent state to the next.
  
  ---
  
  **Input sequences have arbitrary length, which means that LSTMs need to compress information about seen sequence elements**
  
  Rather, the input sequence has an arbitrary length, while the LSTM state vectors have a fixed size. State vectors are the only way for an LSTM to "keep track of important parts". This means that those fixed-size vectors are a bottleneck and there is an information-theoretic upper bound on the amount of information about "important parts" that can be kept by an LSTM.
  
  **LSTMs potentially take multiple decisions. For each decision, something else in the input sequence is most important**
  
  For tasks such as summarization that you mention in the question, an LSTM makes a series of predictions (predicting the tokens of the summary one token at a time). For each prediction, different things in the input sequence might be important. Put another way, for each decision, another *view* of the input may be most helpful.
  
  This is a key motivation for using attention networks. Each time an LSTM is making a prediction, an attention network can provide a dynamic, optimally helpful view of the input sequence.
  
  
  ---
  
  ### Answer by username_5 on 2019-12-27. Score: 0
  
  You can do custom POS Tagging and use it as a multi featured sequence2sequence.
  
  > **Comments:**
  > **username_0**: I think that might still run into the same issues because the comma and positions end up varying based on name so the position isn't as much of a relevant feature  
  
  ---
  
  