Body: <p>When exploring the Twitter Sentiment Analysis dataset on <a href="https://www.kaggle.com/datasets/jp797498e/twitter-entity-sentiment-analysis?select=twitter_training.csv" rel="nofollow noreferrer">Kaggle</a>, I came up with a model that looks like this:</p>
  <pre><code>Model: &quot;sequential&quot;
  _________________________________________________________________
   Layer (type)                Output Shape              Param #   
  =================================================================
   text_vectorization (TextVec  (None, 15)              ***         
   torization)                                                     
                                                                   
   embedding (Embedding)       (None, 15, 200)          ***   
                                                                   
   lstm (LSTM)                 (None, 100, 100)          ***    
                                                                   
   dropout (Dropout)           (None, 100, 100)          ***         
                                                                   
   lstm_1 (LSTM)               (None, 50)                ***     
                                                                   
   dense (Dense)               (None, 3)                 ***       
                                                                   
   dense_1 (Dense)             (None, 3)                 ***       
                                                                   
  =================================================================
  Total params: ***
  Trainable params: ***
  Non-trainable params: ***
  _________________________________________________________________
  </code></pre>
  <p>This model performs okay both in training and on validation (accuracy could reach to 100% for both). However, when I changed the output length of the input sequence in the first layer's (i.e the <code>embedding</code> layer) from 15 to 100, then its performance became worse. In fact, its accuracy could hardly reach 50% and training was also very slow. <a href="https://www.kaggle.com/trankhanhtk/twitter-sentiment-analysis" rel="nofollow noreferrer">Here</a> you can find the updated version of the model and related code.</p>
  <p>Examining the data, I found that roughly 44.5% of training examples have lengths greater than 100.  So I guess it's not unreasonable to pad or truncate sequences to length 100. Also some sources say that we could pad all sequences to the length of the longest sequence in dataset and 100 is clearly not the longest sequence's length in this case.</p>
  <p>So are there some guidelines or best practices for choosing the value for this parameter?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-03-05T16:18:29.710Z
Comments.Text: If you are just changing the output sequence length, its performance drop is understandable because the batch size, learning rate, dropout rate, etc. *was tuned for the 15 words length*. When changing the configuration, we must search new hyperparameters.
Comments.UserDisplayName: Minh-Long Luu
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-03-05T19:30:43.897Z
Comments.Text: This type of issue happens for several other network architectures as well. Whenever I lower down the output sequence length, it always helps speedup convergence & increase accuracy. So my best guess is: it should not relate much to other hyper-parameters.
Comments.UserDisplayName: Tran Khanh
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-03-06T05:15:07.427Z
Comments.Text: I cannot find the paper studying the Sequence length in LSTM, however you can have a look at a similar paper that studied it in the [Attention mechanism](https://arxiv.org/pdf/1409.0473.pdf). Figure 2 showed that the performance generally dropped after 20 words.
Comments.UserDisplayName: Minh-Long Luu
Tags: <tensorflow><sequence-modeling><text-classification>
Title: Increasing "output_sequence_length" in TextVectorization layer worsens model's performance
UserDisplayName: Tran Khanh
ThreadText: # Increasing "output_sequence_length" in TextVectorization layer worsens model's performance
  
  (Asked by: username_0 on 2023-03-05)
  
  When exploring the Twitter Sentiment Analysis dataset on Kaggle, I came up with a model that looks like this:
  Model: "sequential"
  _________________________________________________________________
   Layer (type)                Output Shape              Param #   
  =================================================================
   text_vectorization (TextVec  (None, 15)              ***         
   torization)                                                     
                                                                   
   embedding (Embedding)       (None, 15, 200)          ***   
                                                                   
   lstm (LSTM)                 (None, 100, 100)          ***    
                                                                   
   dropout (Dropout)           (None, 100, 100)          ***         
                                                                   
   lstm_1 (LSTM)               (None, 50)                ***     
                                                                   
   dense (Dense)               (None, 3)                 ***       
                                                                   
   dense_1 (Dense)             (None, 3)                 ***       
                                                                   
  =================================================================
  Total params: ***
  Trainable params: ***
  Non-trainable params: ***
  _________________________________________________________________
  
  This model performs okay both in training and on validation (accuracy could reach to 100% for both). However, when I changed the output length of the input sequence in the first layer's (i.e the embedding layer) from 15 to 100, then its performance became worse. In fact, its accuracy could hardly reach 50% and training was also very slow. Here you can find the updated version of the model and related code.
  Examining the data, I found that roughly 44.5% of training examples have lengths greater than 100.  So I guess it's not unreasonable to pad or truncate sequences to length 100. Also some sources say that we could pad all sequences to the length of the longest sequence in dataset and 100 is clearly not the longest sequence's length in this case.
  So are there some guidelines or best practices for choosing the value for this parameter?
  
  **Tags:** tensorflow, sequence-modeling, text-classification
  > **Comments:**
  > **username_1**: If you are just changing the output sequence length, its performance drop is understandable because the batch size, learning rate, dropout rate, etc. *was tuned for the 15 words length*. When changing the configuration, we must search new hyperparameters.  
  > **username_0**: This type of issue happens for several other network architectures as well. Whenever I lower down the output sequence length, it always helps speedup convergence & increase accuracy. So my best guess is: it should not relate much to other hyper-parameters.  
  > **username_1**: I cannot find the paper studying the Sequence length in LSTM, however you can have a look at a similar paper that studied it in the [Attention mechanism](https://arxiv.org/pdf/1409.0473.pdf). Figure 2 showed that the performance generally dropped after 20 words.  
  
  ---
  
  ## 0 answers
  
  