Answers.Body: <p>I have found nice tutorial in the <code>Tensorflow</code> documentation: <a href="https://www.tensorflow.org/tutorials/structured_data/time_series" rel="nofollow noreferrer">https://www.tensorflow.org/tutorials/structured_data/time_series</a></p>
  <p>They implement and test both strategies.</p>
  <ol>
  <li><p>In the first case, for multi dimensional time series, they output the vector of dimension <code>out_steps * series_dim</code> and then reshape to <code>(out_steps, series dim)</code></p>
  </li>
  <li><p>They create a model (<strong>AR LSTM</strong>), that predicts one step ahead and then apply it several times, where the first step from the previous input is discarded, and the new prediction is last step in new input.</p>
  </li>
  </ol>
  <p><a href="https://i.sstatic.net/JfX6m.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/JfX6m.png" alt="enter image description here" /></a></p>
  <p>Second approach, seems to require less params, but the obtained quality for this specific case, seems to be comparable for both cases.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-09T17:33:33.257Z
Answers.LastActivityDate: 2021-07-09T17:33:33.257Z
Answers.UserDisplayName: spiridon_the_sun_rotator
Body: <p>Suppose one has a time series (univariate or multivariate) and the goal is to predict values of these series several steps ahead. I see two possible strategies:</p>
  <ol>
  <li><p>Create a model (recurrent, convolutional, transformer, whatever) that predicts the value of the signal in the next moment of time, based on the values from previous timestamps from <code>(t_start, t_end)</code>. If we aim to predict not one, but several steps ahead we can pass <code>(signal[t_start + 1: t_end], signal[t_end + 1])</code> to predict <code>signal[t_end + 2]</code> and so on.  In the training stage, we can pass the predicted value of <code>signal[t_end + 1]</code> or the ground truth with some probability, this can be seen as some kind of <em>teacher forcing</em>. In the inference stage, one passes each time the predicted signal. The optimization algorithm aims to minimize (<strong>MSE</strong>, <strong>MAE</strong>) loss between the ground truth and prediction. In other words
  <span class="math-container">$$
  \begin{aligned}
  x_{t+1} &amp;= f(x_t, \ldots, x_{t-N+1}) \\
  x_{t+2} &amp;= f(x_{t+1}, \ldots, x_{t-N+2}) \\
  x_{t+k} &amp;= f(x_{t+1}, \ldots, x_{t-N+k}) \\
  \end{aligned}
  $$</span></p>
  </li>
  <li><p>Create a model that predicts simultaneously several values ahead. Standard layers from DL frameworks (<strong>PyTorch</strong> of <strong>Tensorflow</strong>) for sequence processing problems have two options - output single hidden state in the end or the whole sequence of the hidden states. Therefore, seems like they do not have the functionality, say, to predict values of the time series 16 steps ahead from the values of the last 256 timestamps.
  <span class="math-container">$$
  [y_{t+k}, \ldots, y_{t+1}] = f(x_t, \ldots x_{t - N + 1})
  $$</span>
  I see two potential solutions:</p>
  <ul>
  <li>output hidden state (16) times larger than the expected output and reshape - however, it seems that this approach breaks the locality and causal structure and would not achieve good performance.</li>
  <li>Choose the option, that returns the sequence of the same length as the input (here 256) and take the last (16) tokens of the output. This approach is inapplicable if the length of the prediction exceeds the length of the previous history, but I think, that such long predictions would produce poor quality in any case.</li>
  </ul>
  </li>
  </ol>
  <p>How stock, weather, sales prediction problems are solved usually in practice?</p>
  
Tags: <machine-learning><prediction><time-series><forecasting>
Title: What is a better approach to perform predictions of time-series several values ahead?
UserDisplayName: spiridon_the_sun_rotator
ThreadText: # What is a better approach to perform predictions of time-series several values ahead?
  
  (Asked by: username_0 on 2021-07-08)
  
  Suppose one has a time series (univariate or multivariate) and the goal is to predict values of these series several steps ahead. I see two possible strategies:
  
  1. Create a model (recurrent, convolutional, transformer, whatever) that predicts the value of the signal in the next moment of time, based on the values from previous timestamps from `(t_start, t_end)`. If we aim to predict not one, but several steps ahead we can pass `(signal[t_start + 1: t_end], signal[t_end + 1])` to predict `signal[t_end + 2]` and so on. In the training stage, we can pass the predicted value of `signal[t_end + 1]` or the ground truth with some probability, this can be seen as some kind of *teacher forcing*. In the inference stage, one passes each time the predicted signal. The optimization algorithm aims to minimize (**MSE**, **MAE**) loss between the ground truth and prediction. In other words
     $$
     \begin{aligned}
     x\_{t+1} &= f(x\_t, \ldots, x\_{t-N+1}) \\
     x\_{t+2} &= f(x\_{t+1}, \ldots, x\_{t-N+2}) \\
     x\_{t+k} &= f(x\_{t+1}, \ldots, x\_{t-N+k}) \\
     \end{aligned}
     $$
  2. Create a model that predicts simultaneously several values ahead. Standard layers from DL frameworks (**PyTorch** of **Tensorflow**) for sequence processing problems have two options - output single hidden state in the end or the whole sequence of the hidden states. Therefore, seems like they do not have the functionality, say, to predict values of the time series 16 steps ahead from the values of the last 256 timestamps.
     $$
     [y\_{t+k}, \ldots, y\_{t+1}] = f(x\_t, \ldots x\_{t - N + 1})
     $$
     I see two potential solutions:
  
     * output hidden state (16) times larger than the expected output and reshape - however, it seems that this approach breaks the locality and causal structure and would not achieve good performance.
     * Choose the option, that returns the sequence of the same length as the input (here 256) and take the last (16) tokens of the output. This approach is inapplicable if the length of the prediction exceeds the length of the previous history, but I think, that such long predictions would produce poor quality in any case.
  
  How stock, weather, sales prediction problems are solved usually in practice?
  
  **Tags:** machine-learning, prediction, time-series, forecasting
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2021-07-09. Score: 1
  
  I have found nice tutorial in the `Tensorflow` documentation: <https://www.tensorflow.org/tutorials/structured_data/time_series>
  
  They implement and test both strategies.
  
  1. In the first case, for multi dimensional time series, they output the vector of dimension `out_steps * series_dim` and then reshape to `(out_steps, series dim)`
  2. They create a model (**AR LSTM**), that predicts one step ahead and then apply it several times, where the first step from the previous input is discarded, and the new prediction is last step in new input.
  
  [![enter image description here](https://i.sstatic.net/JfX6m.png)](https://i.sstatic.net/JfX6m.png)
  
  Second approach, seems to require less params, but the obtained quality for this specific case, seems to be comparable for both cases.
  
  
  ---
  
  