Answers.Body: <p>Okay, I finally ended up using a Transformer based Encoder/Decoder model.</p>
  <p>It is a very similar structure as the NLP model described in the <a href="https://arxiv.org/abs/1706.03762" rel="nofollow noreferrer">Attention is all you need</a> paper.</p>
  <p>However, since my use case is not NLP but rather sequential data in general I used to follow the Transformer Encoder/Decoder method as described in this  paper: <a href="https://arxiv.org/pdf/2001.08317.pdf" rel="nofollow noreferrer">Deep Transformer Models for Time Series Forecasting</a></p>
  <p>I slightly modified their model in my implementation to account for my special needs. I used pytorch to build the transformer architecture.</p>
  <p>The results are very good.</p>
  <p>I think when trying to predict sequential data, a encoder/decoder based transformer model will be a good choice and at least worth a look :)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-16T15:51:23.387Z
Answers.LastActivityDate: 2022-10-16T15:51:23.387Z
Answers.UserDisplayName: toom
Answers.Body: <p>A more recent model you can try for sequential data is the <a href="https://github.com/HazyResearch/state-spaces" rel="nofollow noreferrer">S4 model</a>, which is a more lightweight and stable recurrent network that was able to perform better than transformer models on certain tasks. For time series forecasting in particular, there are cases where standard recurrent networks (LSTM, GRU, etc) are still preferable compared to transformer models.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-16T17:04:33.810Z
Answers.LastActivityDate: 2022-10-16T17:04:33.810Z
Answers.UserDisplayName: PeaBrane
Body: <p>I need to predict a binary vector given a sequential dataset meaning the current datapoint depends on its predecessors as well as (known) successors.</p>
  <p>So, it looks something like this:</p>
  <p>Given the sequence: <code>X = [x_1, x_2, x_3, ..., x_N]</code></p>
  <p>I want to predict: <code>Y = [y_1, y_2, y_3, ..., y_N]</code> with <code>y_i \in {0, 1}</code> a binary label</p>
  <p>Now, the sequence <code>X</code> is fully known, that is: future observations in the sequence are completely known at any time.</p>
  <p>Therefore, in contrast to normal time series data, I can also harness any <code>x_i+1, x_i+2,...</code> from the full <code>X</code> sequence for predicting <code>y_i</code> at any time and not only <code>x_i-2, x_i-1, x_i</code> etc.</p>
  <p>Also the data <code>X</code> is a sequence of <code>R^dxN</code> vectors, i.e. N d-dimensional datapoints containing real numbers.</p>
  <p>In my case the dimensionality of the data is <code>d=140</code>.</p>
  <p>Now, what I want to predict is the following:</p>
  <p>What is <code>y_i</code> given <code>X</code>, e.g. what is <code>y_3</code> given the observations <code>x_1, x_2, x_3, x_4, x_5</code>?</p>
  <p>So, eventually I need something like this:</p>
  <pre><code>for i in range(N):
      predict y_i given X = [x_1, x_2, x_3, ..., x_N]
  </code></pre>
  <p>This actually is a many-to-one prediction task.</p>
  <p>Now you could use a RNN, but the problem imho is that &quot;future&quot; observations are not taken into account when applying an RNN.</p>
  <p>Maybe I am wrong about this assumption.</p>
  <p>But therefore I am asking: Which model would you suggest to use for this problem?</p>
  
Tags: <sequence-modeling>
Title: Which model should I apply on sequential data?
UserDisplayName: toom
ThreadText: # Which model should I apply on sequential data?
  
  (Asked by: username_0 on 2022-10-09)
  
  I need to predict a binary vector given a sequential dataset meaning the current datapoint depends on its predecessors as well as (known) successors.
  So, it looks something like this:
  Given the sequence: X = [x_1, x_2, x_3, ..., x_N]
  I want to predict: Y = [y_1, y_2, y_3, ..., y_N] with y_i \in {0, 1} a binary label
  Now, the sequence X is fully known, that is: future observations in the sequence are completely known at any time.
  Therefore, in contrast to normal time series data, I can also harness any x_i+1, x_i+2,... from the full X sequence for predicting y_i at any time and not only x_i-2, x_i-1, x_i etc.
  Also the data X is a sequence of R^dxN vectors, i.e. N d-dimensional datapoints containing real numbers.
  In my case the dimensionality of the data is d=140.
  Now, what I want to predict is the following:
  What is y_i given X, e.g. what is y_3 given the observations x_1, x_2, x_3, x_4, x_5?
  So, eventually I need something like this:
  for i in range(N):
      predict y_i given X = [x_1, x_2, x_3, ..., x_N]
  
  This actually is a many-to-one prediction task.
  Now you could use a RNN, but the problem imho is that "future" observations are not taken into account when applying an RNN.
  Maybe I am wrong about this assumption.
  But therefore I am asking: Which model would you suggest to use for this problem?
  
  **Tags:** sequence-modeling
  
  ---
  
  ## 2 answers
  
  ### Answer by username_0 on 2022-10-16. Score: 0
  
  Okay, I finally ended up using a Transformer based Encoder/Decoder model.
  
  It is a very similar structure as the NLP model described in the [Attention is all you need](https://arxiv.org/abs/1706.03762) paper.
  
  However, since my use case is not NLP but rather sequential data in general I used to follow the Transformer Encoder/Decoder method as described in this paper: [Deep Transformer Models for Time Series Forecasting](https://arxiv.org/pdf/2001.08317.pdf)
  
  I slightly modified their model in my implementation to account for my special needs. I used pytorch to build the transformer architecture.
  
  The results are very good.
  
  I think when trying to predict sequential data, a encoder/decoder based transformer model will be a good choice and at least worth a look :)
  
  
  ---
  
  ### Answer by username_1 on 2022-10-16. Score: 0
  
  A more recent model you can try for sequential data is the [S4 model](https://github.com/HazyResearch/state-spaces), which is a more lightweight and stable recurrent network that was able to perform better than transformer models on certain tasks. For time series forecasting in particular, there are cases where standard recurrent networks (LSTM, GRU, etc) are still preferable compared to transformer models.
  
  
  ---
  
  