Answers.Body: <p>They shouldn't have any issues with short sequences, as short dependencies are easier to learn. The only difficult cases are long dependencies which is where most of the research is geared at. However, this is assuming that by &quot;short sequence&quot; you mean a sequence of text that is fully contained within itself, i.e. there is no cross-sequence dependencies.</p>
  <p>For example, if you have a really long paragraph that doesn't fit in a transformer model, you would have to break that paragraph into many &quot;short sequences&quot;, but each of these sequence may have a dependency that depends on another sequence, i.e. cross-sequence dependencies. For these cross-sequence dependencies, any model with recurrence should do better than ones without (e.g. RNN, LSTM, Transformer-XL).</p>
  <p>If each short sequence is self-contained, then all of the models should perform pretty well.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-19T20:42:52.143Z
Answers.Text: Thank you very much for the explanation. I am not really dealing with NLP, but with sequences which have a reasonably similar representation. All those sequences are self-contained. My idea was to exploit attention (I was using plain LSTMs, with quite good performance) to leverage the fact that I know that some elements of my input sequence are more relevant to predict certain elements of the output sequence (the problem is encoded as a seq2seq one). So, in this context, attention, even if I am dealing with short sequences, could be helpful?
Answers.UserDisplayName: nsacco
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-19T22:47:53.117Z
Answers.Text: Yes, if your task parallels a seq2seq task, then it is very likely Transformers will do better than LSTM on ideal conditions. But if you have a small transformer model, small dataset, or small hardware, transformers will likely to perform worse than LSTM. The reason transformers push state of the art is because companies can train extremely huge models on extremely huge datasets very quickly. But if you are training a model from scratch and you do not have powerful hardware and millions of data points, I think LSTM would be more practical.
Answers.UserDisplayName: user3667125
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-18T21:32:53.003Z
Answers.LastActivityDate: 2020-12-18T21:32:53.003Z
Answers.UserDisplayName: user3667125
Body: <p>I am aware that the attention mechanism can be used to deal with long sequences, where problems related to gradient vanishing and, more generally, representing effectively the whole sequence arise.</p>
  <p>However, I was wondering if attention, applied either to seq2seq RNN/GRU/LSTM or via Transformers, can contribute to improving the overall performance (as well as giving some sort of interpretability through the attention weights?) in the case of relatively short sequences (let's say around 20-30 elements each).</p>
  
Tags: <deep-learning><natural-language-processing><transformer><attention><performance>
Title: Can the attention mechanism improve the performance in the case of short sequences?
UserDisplayName: nsacco
ThreadText: # Can the attention mechanism improve the performance in the case of short sequences?
  
  (Asked by: username_0 on 2020-12-17)
  
  I am aware that the attention mechanism can be used to deal with long sequences, where problems related to gradient vanishing and, more generally, representing effectively the whole sequence arise.
  
  However, I was wondering if attention, applied either to seq2seq RNN/GRU/LSTM or via Transformers, can contribute to improving the overall performance (as well as giving some sort of interpretability through the attention weights?) in the case of relatively short sequences (let's say around 20-30 elements each).
  
  **Tags:** deep-learning, natural-language-processing, transformer, attention, performance
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-12-18. Score: 1
  
  They shouldn't have any issues with short sequences, as short dependencies are easier to learn. The only difficult cases are long dependencies which is where most of the research is geared at. However, this is assuming that by "short sequence" you mean a sequence of text that is fully contained within itself, i.e. there is no cross-sequence dependencies.
  
  For example, if you have a really long paragraph that doesn't fit in a transformer model, you would have to break that paragraph into many "short sequences", but each of these sequence may have a dependency that depends on another sequence, i.e. cross-sequence dependencies. For these cross-sequence dependencies, any model with recurrence should do better than ones without (e.g. RNN, LSTM, Transformer-XL).
  
  If each short sequence is self-contained, then all of the models should perform pretty well.
  
  > **Comments:**
  > **username_0**: Thank you very much for the explanation. I am not really dealing with NLP, but with sequences which have a reasonably similar representation. All those sequences are self-contained. My idea was to exploit attention (I was using plain LSTMs, with quite good performance) to leverage the fact that I know that some elements of my input sequence are more relevant to predict certain elements of the output sequence (the problem is encoded as a seq2seq one). So, in this context, attention, even if I am dealing with short sequences, could be helpful?  
  > **username_1**: Yes, if your task parallels a seq2seq task, then it is very likely Transformers will do better than LSTM on ideal conditions. But if you have a small transformer model, small dataset, or small hardware, transformers will likely to perform worse than LSTM. The reason transformers push state of the art is because companies can train extremely huge models on extremely huge datasets very quickly. But if you are training a model from scratch and you do not have powerful hardware and millions of data points, I think LSTM would be more practical.  
  
  ---
  
  