Body: <p>ETA: More concise wording: Why do some implementations use batches of data taken from within the same sequence? Does this not make the cell state useless?</p>
  <p>Using the example of an LSTM, it has a <strong>hidden state</strong> and <strong>cell state</strong>. These states are updated as new inputs are passed to an LSTM. The problem here, is if you use batches of data taken from the same timeframe, the <strong>hidden state</strong> and <strong>cell state</strong> can't be computed on previous values.</p>
  <h1>This is a major problem. The main advantage of an LSTM is this exact mechanic</h1>
  <p>For an example of what I mean, <a href="https://www.kaggle.com/code/carlmcbrideellis/lstm-time-series-prediction-sine-wave-example/notebook" rel="nofollow noreferrer">take the following</a>. A simple LSTM used to predict a sine wave. At training time, I can't think of any way you could use batches. As you have one timeseries here that you are predicting, you <strong>have</strong> to start at time step 0 in order to properly compute and train the <strong>hidden state</strong> and <strong>cell state</strong>.</p>
  <p>Taking batches like in the image below and computing them in parallel will mean the <strong>hidden</strong> and <strong>cell state</strong> can't be computed properly.</p>
  <p><a href="https://i.sstatic.net/ANreT.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/ANreT.png" alt="enter image description here" /></a></p>
  <p>And yet, in the example I gave the <strong>batch_size</strong> is set to 10? This makes no sense to me. It also doesn't help tensorflow syntax isn't exactly the most verbose...</p>
  <p>The only use case for batches in an LSTM I can see is if you have multiple totally independent sets of timeseries that can all be computed from timestep 0 in parallel with each having it's own <strong>cell</strong> and <strong>hidden state</strong></p>
  <h2>My implementation</h2>
  <p>I actually duplicated the example LSTM from above but used Pytorch instead. <a href="https://www.kaggle.com/code/seangribben/lstm-time-series-prediction-pytorch-sine-wave" rel="nofollow noreferrer">My code can be found in a Kaggle notebook here</a>, but as you can see, I've commented out the <code>LSTM</code> from the model and replaced it with a <code>fc</code> layer which performs just as well as the <code>LSTM</code>, because like I said, while using batches in this way it makes the LSTM utterly redundant.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-20T16:51:40.370Z
Comments.Text: Can you please elaborate on the problem you're having with the inability to use batches for sequences (using a LSTM)?
Comments.UserDisplayName: hal9000
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-20T16:58:03.107Z
Comments.Text: So you have a batch of sequences , and let assume you start with an zero's cell state. This batch of sequences you can break up into normal batches of inputs, the cellstate then binds all these batches back together into a sequence through the computational graph. So for as far as I know you shouldn't run into any problems. :p
Comments.UserDisplayName: hal9000
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-25T01:54:05.830Z
Comments.Text: @hal9000 Your wording's a little confusing, but I think what you mean is having each element of a batch be an independent sequence? In which case I understand that's a possibility, but the example I gave has one sequence that is broken into batches, which is what doesn't make sense to me
Comments.UserDisplayName: Recessive
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-26T16:30:59.863Z
Comments.Text: Just having one sequence then breaking the sequence up so each part doesn't rely on other parts isn't how an recurrent neural network works, I totally agree with you that wouldn't work.
Comments.UserDisplayName: hal9000
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-26T16:32:12.153Z
Comments.Text: But you can create from one sequence a lot of  sequences, each with different starting points and make a batch out of that. Maybe that is what is meant with it??
Comments.UserDisplayName: hal9000
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-26T16:33:20.163Z
Comments.Text: yes, I meant each element  of the batch to be an independent sequence. (then maybe create a bunch of sequences from this one sequence (having random starting points for example))
Comments.UserDisplayName: hal9000
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-27T01:57:53.080Z
Comments.Text: @hal9000 I guess that would depend on how dependent each segment within a sequence is to another segment? Like splitting a paragraph up into smaller sentences/words would completely ruin the networks ability to learn the overall meaning of the paragraph, but in the example the sine wave is repetitive so it would probably work to take random splits
Comments.UserDisplayName: Recessive
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-28T10:26:59.987Z
Comments.Text: That is true, for text you could for example add noise to the embedding of some words so it has to infer those words from the context when doing it's prediction. (where your batch could be all kind of different combinations of words knocked out)
Comments.UserDisplayName: hal9000
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-28T10:29:53.007Z
Comments.Text: preferably you would have multiple texts with their problem of-course assuming it's a supervised task, and have a combination of different texts in each batch.
Comments.UserDisplayName: hal9000
Tags: <machine-learning><recurrent-neural-networks><long-short-term-memory><pytorch><batch-learning>
Title: How is it possible to use batches of data from within the same sequence with an LSTM?
UserDisplayName: Recessive
ThreadText: # How is it possible to use batches of data from within the same sequence with an LSTM?
  
  (Asked by: username_0 on 2022-07-20)
  
  ETA: More concise wording: Why do some implementations use batches of data taken from within the same sequence? Does this not make the cell state useless?
  
  Using the example of an LSTM, it has a **hidden state** and **cell state**. These states are updated as new inputs are passed to an LSTM. The problem here, is if you use batches of data taken from the same timeframe, the **hidden state** and **cell state** can't be computed on previous values.
  
  # This is a major problem. The main advantage of an LSTM is this exact mechanic
  
  For an example of what I mean, [take the following](https://www.kaggle.com/code/carlmcbrideellis/lstm-time-series-prediction-sine-wave-example/notebook). A simple LSTM used to predict a sine wave. At training time, I can't think of any way you could use batches. As you have one timeseries here that you are predicting, you **have** to start at time step 0 in order to properly compute and train the **hidden state** and **cell state**.
  
  Taking batches like in the image below and computing them in parallel will mean the **hidden** and **cell state** can't be computed properly.
  
  [![enter image description here](https://i.sstatic.net/ANreT.png)](https://i.sstatic.net/ANreT.png)
  
  And yet, in the example I gave the **batch\_size** is set to 10? This makes no sense to me. It also doesn't help tensorflow syntax isn't exactly the most verbose...
  
  The only use case for batches in an LSTM I can see is if you have multiple totally independent sets of timeseries that can all be computed from timestep 0 in parallel with each having it's own **cell** and **hidden state**
  
  ## My implementation
  
  I actually duplicated the example LSTM from above but used Pytorch instead. [My code can be found in a Kaggle notebook here](https://www.kaggle.com/code/seangribben/lstm-time-series-prediction-pytorch-sine-wave), but as you can see, I've commented out the `LSTM` from the model and replaced it with a `fc` layer which performs just as well as the `LSTM`, because like I said, while using batches in this way it makes the LSTM utterly redundant.
  
  **Tags:** machine-learning, recurrent-neural-networks, long-short-term-memory, pytorch, batch-learning
  
  > **Comments:**
  > **username_1**: Can you please elaborate on the problem you're having with the inability to use batches for sequences (using a LSTM)?  
  > **username_1**: So you have a batch of sequences , and let assume you start with an zero's cell state. This batch of sequences you can break up into normal batches of inputs, the cellstate then binds all these batches back together into a sequence through the computational graph. So for as far as I know you shouldn't run into any problems. :p  
  > **username_0**: @username_1 Your wording's a little confusing, but I think what you mean is having each element of a batch be an independent sequence? In which case I understand that's a possibility, but the example I gave has one sequence that is broken into batches, which is what doesn't make sense to me  
  > **username_1**: Just having one sequence then breaking the sequence up so each part doesn't rely on other parts isn't how an recurrent neural network works, I totally agree with you that wouldn't work.  
  > **username_1**: But you can create from one sequence a lot of  sequences, each with different starting points and make a batch out of that. Maybe that is what is meant with it??  
  > **username_1**: yes, I meant each element  of the batch to be an independent sequence. (then maybe create a bunch of sequences from this one sequence (having random starting points for example))  
  > **username_0**: @username_1 I guess that would depend on how dependent each segment within a sequence is to another segment? Like splitting a paragraph up into smaller sentences/words would completely ruin the networks ability to learn the overall meaning of the paragraph, but in the example the sine wave is repetitive so it would probably work to take random splits  
  > **username_1**: That is true, for text you could for example add noise to the embedding of some words so it has to infer those words from the context when doing it's prediction. (where your batch could be all kind of different combinations of words knocked out)  
  > **username_1**: preferably you would have multiple texts with their problem of-course assuming it's a supervised task, and have a combination of different texts in each batch.  
  
  ---
  
  ## 0 answers
  
  