Answers.Body: <p>How about a <a href="https://arxiv.org/abs/1611.05267" rel="nofollow noreferrer">Temporal Convolutional Network</a>? It feels like for such a long sequences having the recurrent/memory based approach is not too feasible. But, intuitively, the 1D convolutions should be able to pick out those rare features from your extremely long sequences.</p>
  <p>There are also claims that <a href="https://arxiv.org/abs/1803.01271" rel="nofollow noreferrer">TCNs are comparable to RNNs</a> in performance on common tasks, so there's that.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-13T17:40:11.343Z
Answers.LastActivityDate: 2021-04-13T17:40:11.343Z
Answers.UserDisplayName: Kostya
Body: <p>I want an LSTM to output one of two classes (Y, N), per frame, based on all the input so far.<br />
  My original inputs are very long (~100000 samples long, far more than a standard LSTM training can handle due to vanishing gradients).</p>
  <ol>
  <li>If the last seen instance out of the tokens (A, B) was A, output Y.</li>
  <li>If the last seen instance out of the tokens (A, B) was B, output N.</li>
  <li>The very long sequence is guaranteed to start with either A or B.</li>
  </ol>
  <p>If the sequence was short, this would be quite easy.</p>
  <p>For example, the following top lines and bottom lines correspond to inputs and required outputs:</p>
  <pre><code>ABCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
  YNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN
  
  ACCCCCCCCCBCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCACCCCCCCCCCCCCCCCACCCCCCCC
  YYYYYYYYYYNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNYYYYYYYYYYYYYYYYYYYYYYYYYY
  </code></pre>
  <hr />
  <p>Looks easy enough, just push batches comprised of chunks of the long sequence to the LSTM and have a coffee, right?<br />
  However, for my case, the available inputs are (A, B, C), of which (A, B) are extremely rare, meaning I can have batches comprised of 100% C's. The LSTM has no chance then, if not fed with some current state, telling it about the last A or B seen.<br />
  Unfortunately, this &quot;state&quot; is really something learned, and I can't just feed it as input AFAIK.</p>
  <pre><code>CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
  ????????????????????????????????????????????????????????????????????
  </code></pre>
  <hr />
  <p>I am looking for a standard practice, or other references on how to train an LSTM or other RNN based model to be able to classify based on rare events far in history.</p>
  <p>I hope this is clear, if not please ask and I will edit.</p>
  <hr />
  <p>Please note that the data is labeled, and labeling can't be generated automatically for this task. The above is just an example for ease of understanding, the reality is more complicated.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-13T15:57:04.740Z
Comments.Text: Why do you need an LSTM here? Is either A or B a composite event over multiple time steps? If not, then what is wrong with training a classifier for A, B, C and applying your simple logic for Y/N over a basic variable? There may still be an issue due to unbalanced cases for A/B/C, but it seems more tractable than throwing an LSTM at the problem if you don't actually care about anything subtle over many timestamps (i.e. the Cs are all uninteresting and do not affect eventual classification of A or B), just the classification of individual events.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-13T16:03:20.793Z
Comments.Text: @NeilSlater The events are indeed composite. But even if they weren't, I am not following how you propose not using some sort of a model with memory. By feeding back that classification variable into the network? Isn't that LSTM with less steps [which may be ok, just making sure]?
Comments.UserDisplayName: Gulzar
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-13T16:05:19.823Z
Comments.Text: More than the A/B events are composite, the logic for closing the loop between Y/N to A/B is composite. For example, the switch may not be immediate as stated in the question.
Comments.UserDisplayName: Gulzar
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-13T17:00:27.077Z
Comments.Text: My point is that your "model with a memory" can literally be some Python (or whatever) var `last_seen = 'A'`. Unless there is some subtlety and some Cs are different from other Cs (in terms of whether or not A or B eventually happens), then forcing the problem to be "I must write an LSTM with very long memory" might be making a rod for your own back. It is hard to tell because you have simplified things, which is fine. However, i don't see that the Y/N switch is "composite" - the rules you state are simple and absolute, and there is no statement in any other direction.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-13T21:45:11.147Z
Comments.Text: @Neil Slater alright, this is fair. I will haveto come up with something a bit more elaborated then. I will sleep on it.
Comments.UserDisplayName: Gulzar
Tags: <deep-learning><reference-request><recurrent-neural-networks><long-short-term-memory><time-series>
Title: How to train an LSTM to classify based on rare historic event?
UserDisplayName: Gulzar
ThreadText: # How to train an LSTM to classify based on rare historic event?
  
  (Asked by: username_0 on 2021-04-13)
  
  I want an LSTM to output one of two classes (Y, N), per frame, based on all the input so far.  
  My original inputs are very long (~100000 samples long, far more than a standard LSTM training can handle due to vanishing gradients).
  
  1. If the last seen instance out of the tokens (A, B) was A, output Y.
  2. If the last seen instance out of the tokens (A, B) was B, output N.
  3. The very long sequence is guaranteed to start with either A or B.
  
  If the sequence was short, this would be quite easy.
  
  For example, the following top lines and bottom lines correspond to inputs and required outputs:
  
  ```
  ABCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
  YNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN
  
  ACCCCCCCCCBCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCACCCCCCCCCCCCCCCCACCCCCCCC
  YYYYYYYYYYNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNYYYYYYYYYYYYYYYYYYYYYYYYYY
  
  ```
  
  ---
  
  Looks easy enough, just push batches comprised of chunks of the long sequence to the LSTM and have a coffee, right?  
  However, for my case, the available inputs are (A, B, C), of which (A, B) are extremely rare, meaning I can have batches comprised of 100% C's. The LSTM has no chance then, if not fed with some current state, telling it about the last A or B seen.  
  Unfortunately, this "state" is really something learned, and I can't just feed it as input AFAIK.
  
  ```
  CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
  ????????????????????????????????????????????????????????????????????
  
  ```
  
  ---
  
  I am looking for a standard practice, or other references on how to train an LSTM or other RNN based model to be able to classify based on rare events far in history.
  
  I hope this is clear, if not please ask and I will edit.
  
  ---
  
  Please note that the data is labeled, and labeling can't be generated automatically for this task. The above is just an example for ease of understanding, the reality is more complicated.
  
  **Tags:** deep-learning, reference-request, recurrent-neural-networks, long-short-term-memory, time-series
  
  > **Comments:**
  > **username_1**: Why do you need an LSTM here? Is either A or B a composite event over multiple time steps? If not, then what is wrong with training a classifier for A, B, C and applying your simple logic for Y/N over a basic variable? There may still be an issue due to unbalanced cases for A/B/C, but it seems more tractable than throwing an LSTM at the problem if you don't actually care about anything subtle over many timestamps (i.e. the Cs are all uninteresting and do not affect eventual classification of A or B), just the classification of individual events.  
  > **username_0**: @username_1 The events are indeed composite. But even if they weren't, I am not following how you propose not using some sort of a model with memory. By feeding back that classification variable into the network? Isn't that LSTM with less steps [which may be ok, just making sure]?  
  > **username_0**: More than the A/B events are composite, the logic for closing the loop between Y/N to A/B is composite. For example, the switch may not be immediate as stated in the question.  
  > **username_1**: My point is that your "model with a memory" can literally be some Python (or whatever) var `last_seen = 'A'`. Unless there is some subtlety and some Cs are different from other Cs (in terms of whether or not A or B eventually happens), then forcing the problem to be "I must write an LSTM with very long memory" might be making a rod for your own back. It is hard to tell because you have simplified things, which is fine. However, i don't see that the Y/N switch is "composite" - the rules you state are simple and absolute, and there is no statement in any other direction.  
  > **username_0**: @username_1 alright, this is fair. I will haveto come up with something a bit more elaborated then. I will sleep on it.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-04-13. Score: 1
  
  How about a [Temporal Convolutional Network](https://arxiv.org/abs/1611.05267)? It feels like for such a long sequences having the recurrent/memory based approach is not too feasible. But, intuitively, the 1D convolutions should be able to pick out those rare features from your extremely long sequences.
  
  There are also claims that [TCNs are comparable to RNNs](https://arxiv.org/abs/1803.01271) in performance on common tasks, so there's that.
  
  
  ---
  
  