Answers.Body: <p>You could sequentially pass in each element of your sequential data and save the hidden and cell states in a separate buffer. In a typical LSTM implementation, you input the entire sequence and the hidden and cell states are propagated internally. In the end, the final hidden and cell states returned as the output. This works if your input is all the same length. Instead, you can handle sequentially giving the next element to the LSTM as well as the hidden and cell state yourself. To keep it efficient you can batch your inputs by the batch dimension (batch_first=True in the <a href="https://pytorch.org/docs/master/generated/torch.nn.LSTM.html" rel="nofollow noreferrer">pytorch LSTM implementation</a>).</p>
  <p>For example, propose you have 5 sequences of length 5, 4, 3, 2, and 1. You initialize your hidden and cell state buffers for each of the sequences and pass the first batch containing the first element of all 5 sequences. You save the output hidden and cell states in the buffers for each sequence. Next, you input the batch of the second element of the 4 sequences with length &gt; 1, and save the states in the respective sequence buffers and so and so forth until you exhaust the sequence of greatest length.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-30T09:11:35.640Z
Answers.Text: What if all of the sequences are different length, so I can not batch the same length sequence together?
Answers.UserDisplayName: Swakshar Deb
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-30T09:15:10.973Z
Answers.Text: You don't need to batch the sequences, you just need to batch the elements of the sequences. Batch all the first elements of your sequences into one vector (or matrix if your elements are multidimensional), all the second elements in another etc.
Answers.UserDisplayName: Ferdinand Schlatt
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-30T09:28:06.817Z
Answers.Text: I do not want to pass the first batch containing the first element of all 5 sequences like this, I am also curious about why one should pass input like this?
Answers.UserDisplayName: Swakshar Deb
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-30T09:34:03.373Z
Answers.Text: Because this avoids having to pad your input to efficiently process variable length sequences. AFAIK the LSTM implementations in all typical python packages receive input of size Batch x Sequence x Hidden (or a permutation). Since your sequence dimension is not the same for every sequence, you either need to pad your sequences. Or as suggested in the answer, you can change the sequence dimension to 1, but need to handle the hidden and cell states of the LSTM yourself.
Answers.UserDisplayName: Ferdinand Schlatt
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-30T08:48:25.230Z
Answers.LastActivityDate: 2020-07-30T09:08:12.317Z
Answers.LastEditDate: 2020-07-30T09:08:12.317Z
Answers.UserDisplayName: Ferdinand Schlatt
Answers.Body: <p>This question might be deep learning framework specific, so my answer is intended for Keras.</p>
  <p>In keras, an LSTM cell accept 3D tensor, which is <code>(num of samples, num of timesteps, num of features)</code>. The question is about varying num of timesteps.</p>
  <p>My naive approach will be like this:</p>
  <ol>
  <li>Define model architecture in a function that accept num of timestep as parameter that returning model object</li>
  <li>Prepare next sample</li>
  <li>Instantiate a model with num of timestep based on current sample</li>
  <li>Load the latest weights if exist</li>
  <li>Train one epoch</li>
  <li>Save the latest weights</li>
  <li>Back to number 2</li>
  </ol>
  <p>So, the code  will be looks like this:</p>
  <pre><code>from keras.models import Model
  from keras.layers import LSTM, Input, Dense
  import numpy as np
  
  def my_robust_lstm_model(num_of_timesteps=512, num_of_features=1):
      input_layer = Input(shape=(num_of_timesteps, num_of_features))
      hidden_layer = LSTM(64)(input_layer)
      output_layer = Dense(1)(hidden_layer)  # Example output layer
      model = Model(inputs=input_layer, outputs=output_layer)
      return model
  
  def array_generator(my_list_pickle_path):
      # Implement how one sample X_train, y_train looks like that has varying length
      # Example implementation: (Replace this with actual data generation logic)
      while True:
          X_train = np.random.random((1, 10, 1))  # Dummy data with varying lengths
          y_train = np.random.random((1, 1))
          yield X_train, y_train
  
  generator = array_generator('path/to/pickle/file')
  EPOCHS = 10
  
  for _ in range(EPOCHS):
      current_X, current_y = next(generator)
      num_of_timesteps = current_X.shape[1]
      num_of_features = current_X.shape[2]
      current_model = my_robust_lstm_model(num_of_timesteps=num_of_timesteps, num_of_features=num_of_features)
      
      try:
          current_model.load_weights('latest.weights.h5')
      except Exception as e:
          print(f'Weights don\'t exist or cannot be loaded: {e}, skipping')
      
      current_model.compile(optimizer='adam', loss='mse')  # Example compile parameters
      current_model.fit(current_X, current_y, batch_size=1, epochs=1)
      current_model.save_weights('latest.weights.h5')
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-03T12:37:19.483Z
Answers.LastActivityDate: 2024-09-03T12:44:38.887Z
Answers.LastEditDate: 2024-09-03T12:44:38.887Z
Answers.UserDisplayName: Muhammad Ikhwan Perwira
Body: <p>I have a dataset where each of the training instances is different in the length and the data is sequential. So, I design an LSTM but I am thinking about how to train the LSTM. In fixed-length data, we just keep all of the input in an array and pass it to the network, but here the case is different. I can not store varying length data in an array and I do not want to use padding to make it fixed length. Now, should I train the LSTM where each training instance are varying in length?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-30T08:19:04.720Z
Comments.Text: Are you predicting a sequence e.g. like in machine translation?
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-30T08:22:49.300Z
Comments.Text: I am performing prediction but not like the machine translation
Comments.UserDisplayName: Swakshar Deb
Tags: <deep-learning><natural-language-processing><tensorflow>
Title: How to train an LSTM with varying length input?
UserDisplayName: Swakshar Deb
ThreadText: # How to train an LSTM with varying length input?
  
  (Asked by: username_0 on 2020-07-30)
  
  I have a dataset where each of the training instances is different in the length and the data is sequential. So, I design an LSTM but I am thinking about how to train the LSTM. In fixed-length data, we just keep all of the input in an array and pass it to the network, but here the case is different. I can not store varying length data in an array and I do not want to use padding to make it fixed length. Now, should I train the LSTM where each training instance are varying in length?
  
  **Tags:** deep-learning, natural-language-processing, tensorflow
  
  > **Comments:**
  > **username_1**: Are you predicting a sequence e.g. like in machine translation?  
  > **username_0**: I am performing prediction but not like the machine translation  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2020-07-30. Score: 2
  
  You could sequentially pass in each element of your sequential data and save the hidden and cell states in a separate buffer. In a typical LSTM implementation, you input the entire sequence and the hidden and cell states are propagated internally. In the end, the final hidden and cell states returned as the output. This works if your input is all the same length. Instead, you can handle sequentially giving the next element to the LSTM as well as the hidden and cell state yourself. To keep it efficient you can batch your inputs by the batch dimension (batch\_first=True in the [pytorch LSTM implementation](https://pytorch.org/docs/master/generated/torch.nn.LSTM.html)).
  
  For example, propose you have 5 sequences of length 5, 4, 3, 2, and 1. You initialize your hidden and cell state buffers for each of the sequences and pass the first batch containing the first element of all 5 sequences. You save the output hidden and cell states in the buffers for each sequence. Next, you input the batch of the second element of the 4 sequences with length > 1, and save the states in the respective sequence buffers and so and so forth until you exhaust the sequence of greatest length.
  
  > **Comments:**
  > **username_0**: What if all of the sequences are different length, so I can not batch the same length sequence together?  
  > **username_2**: You don't need to batch the sequences, you just need to batch the elements of the sequences. Batch all the first elements of your sequences into one vector (or matrix if your elements are multidimensional), all the second elements in another etc.  
  > **username_0**: I do not want to pass the first batch containing the first element of all 5 sequences like this, I am also curious about why one should pass input like this?  
  > **username_2**: Because this avoids having to pad your input to efficiently process variable length sequences. AFAIK the LSTM implementations in all typical python packages receive input of size Batch x Sequence x Hidden (or a permutation). Since your sequence dimension is not the same for every sequence, you either need to pad your sequences. Or as suggested in the answer, you can change the sequence dimension to 1, but need to handle the hidden and cell states of the LSTM yourself.  
  
  ---
  
  ### Answer by username_3 on 2024-09-03. Score: 0
  
  This question might be deep learning framework specific, so my answer is intended for Keras.
  
  In keras, an LSTM cell accept 3D tensor, which is `(num of samples, num of timesteps, num of features)`. The question is about varying num of timesteps.
  
  My naive approach will be like this:
  
  1. Define model architecture in a function that accept num of timestep as parameter that returning model object
  2. Prepare next sample
  3. Instantiate a model with num of timestep based on current sample
  4. Load the latest weights if exist
  5. Train one epoch
  6. Save the latest weights
  7. Back to number 2
  
  So, the code will be looks like this:
  
  ```
  from keras.models import Model
  from keras.layers import LSTM, Input, Dense
  import numpy as np
  
  def my_robust_lstm_model(num_of_timesteps=512, num_of_features=1):
      input_layer = Input(shape=(num_of_timesteps, num_of_features))
      hidden_layer = LSTM(64)(input_layer)
      output_layer = Dense(1)(hidden_layer)  # Example output layer
      model = Model(inputs=input_layer, outputs=output_layer)
      return model
  
  def array_generator(my_list_pickle_path):
      # Implement how one sample X_train, y_train looks like that has varying length
      # Example implementation: (Replace this with actual data generation logic)
      while True:
          X_train = np.random.random((1, 10, 1))  # Dummy data with varying lengths
          y_train = np.random.random((1, 1))
          yield X_train, y_train
  
  generator = array_generator('path/to/pickle/file')
  EPOCHS = 10
  
  for _ in range(EPOCHS):
      current_X, current_y = next(generator)
      num_of_timesteps = current_X.shape[1]
      num_of_features = current_X.shape[2]
      current_model = my_robust_lstm_model(num_of_timesteps=num_of_timesteps, num_of_features=num_of_features)
      
      try:
          current_model.load_weights('latest.weights.h5')
      except Exception as e:
          print(f'Weights don\'t exist or cannot be loaded: {e}, skipping')
      
      current_model.compile(optimizer='adam', loss='mse')  # Example compile parameters
      current_model.fit(current_X, current_y, batch_size=1, epochs=1)
      current_model.save_weights('latest.weights.h5')
  
  ```
  
  
  ---
  
  