Answers.Body: <p>I have not come across a model like this yet. BUT</p>
  
  <p>If you have not tried smaller models I'd recommend trying that first.</p>
  
  <p>Justification: this lets you use <a href="https://machinelearningmastery.com/learning-curves-for-diagnosing-machine-learning-model-performance/" rel="nofollow noreferrer">learning curves</a> to diagnose what to do next. </p>
  
  <p>Also, you might try starting with a GRU (the LSTM overhead may not be needed).</p>
  
  <p><strong>One Idea for a starting model</strong></p>
  
  <p>Observe that turnout <strong>does not</strong> affect weather. So the weather forecast model will be independent of the turnout model but not vice versa.</p>
  
  <p>(Note: I'm using RNN to denote which ever recurrent architecture you choose)</p>
  
  <p><strong>Formulation for simultaneous prediction</strong>:</p>
  
  <p>Weather</p>
  
  <p><span class="math-container">$\text{RNN}_w$</span> is your weather prediction model</p>
  
  <p><span class="math-container">$\hat w_t = \text{RNN}_w(\hat w_{t-1})$</span> be the predicted weather at time <span class="math-container">$t$</span></p>
  
  <p>Turnout: The idea here is that people will or will not go to the beach for various reasons (overcrowded, too hot or stormy etc). So in the beach population prediction task we use all these as <strong>features</strong> to predict the population at the next time-step. This reduces the problem to any one of the classical models already developed. </p>
  
  <p><span class="math-container">$\text{RNN}_p$</span> is your turnout (beach population) prediction model</p>
  
  <p><span class="math-container">$\hat p_t$</span> be the predicted turnout at time <span class="math-container">$t$</span></p>
  
  <p><span class="math-container">$\hat c_t=[\hat p_t, \hat w_t]$</span> is the concatenation at time t</p>
  
  <p><span class="math-container">$\hat p_{t+1} = \text{RNN}_p(\hat c_t)$</span></p>
  
  <p><strong>Formulation in the presence of weather forecast</strong>:</p>
  
  <p>Simply replace <span class="math-container">$\hat w_t$</span> with the true weather forecasts <span class="math-container">$\hat f_t$</span>.</p>
  
  <p><strong>A final warning</strong></p>
  
  <p>Unless you think your RNN is better at weather prediction than the forecasting systems I would <strong>not</strong> recommend using <span class="math-container">$\text{RNN}_w$</span> in a production application.</p>
  
  <p>I hope this helps.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-02T16:30:58.593Z
Answers.LastActivityDate: 2019-09-02T16:42:25.703Z
Answers.LastEditDate: 2019-09-02T16:42:25.703Z
Answers.UserDisplayName: respectful
Body: <p>There is plenty of literature describing LSTMs in a lot of detail and how to use them for multi-variate or uni-variate forecasting problems. What I couldn't find though, is any papers or discussions describing time series forecasting where we have correlated forecast data.</p>
  
  <p>An example is best to describe what I mean. Say I wanted to predict number of people at a beach for the next 24 hours and I want to predict this in hourly granularity. This quantity of people would clearly depend on the past quantity of people at the beach as well as the weather. Now I can make an LSTM architecture of some sort to predict these future quantities based upon what happened in the past quite easily. But what if I now have access to weather forecasts for the next 24 hours too? (and historical forecast data too).</p>
  
  <p>The architecture I came up with looks like this:</p>
  
  <p><a href="https://i.sstatic.net/Eqegn.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Eqegn.png" alt="Forecast architecture"></a></p>
  
  <p>So I train the left upper branch on forecast data, then train the right upper branch on out-turn data, then freeze their layers to and join them to form the final network in the picture and train that on both forecasts and out-turns. (when I say train, the output is always the forecast for the next 24 hours). This method does in fact have better performance than using forecasts or out-turns alone.</p>
  
  <p>I guess my question is, has anyone seen any literature around on this topic and/ or knows a better way to solve these sort of multivariate time series forecasting problems and is my method okay or completely flawed?</p>
  
Tags: <deep-learning><ai-design><long-short-term-memory><time-series>
Title: How should I design the LSTM architecture for multivariate time series forecasting problems?
UserDisplayName: Dema Ushchapovskyy
ThreadText: # How should I design the LSTM architecture for multivariate time series forecasting problems?
  
  (Asked by: username_0 on 2019-05-23)
  
  There is plenty of literature describing LSTMs in a lot of detail and how to use them for multi-variate or uni-variate forecasting problems. What I couldn't find though, is any papers or discussions describing time series forecasting where we have correlated forecast data.
  
  An example is best to describe what I mean. Say I wanted to predict number of people at a beach for the next 24 hours and I want to predict this in hourly granularity. This quantity of people would clearly depend on the past quantity of people at the beach as well as the weather. Now I can make an LSTM architecture of some sort to predict these future quantities based upon what happened in the past quite easily. But what if I now have access to weather forecasts for the next 24 hours too? (and historical forecast data too).
  
  The architecture I came up with looks like this:
  
  [![Forecast architecture](https://i.sstatic.net/Eqegn.png)](https://i.sstatic.net/Eqegn.png)
  
  So I train the left upper branch on forecast data, then train the right upper branch on out-turn data, then freeze their layers to and join them to form the final network in the picture and train that on both forecasts and out-turns. (when I say train, the output is always the forecast for the next 24 hours). This method does in fact have better performance than using forecasts or out-turns alone.
  
  I guess my question is, has anyone seen any literature around on this topic and/ or knows a better way to solve these sort of multivariate time series forecasting problems and is my method okay or completely flawed?
  
  **Tags:** deep-learning, ai-design, long-short-term-memory, time-series
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-09-02. Score: 1
  
  I have not come across a model like this yet. BUT
  
  If you have not tried smaller models I'd recommend trying that first.
  
  Justification: this lets you use [learning curves](https://machinelearningmastery.com/learning-curves-for-diagnosing-machine-learning-model-performance/) to diagnose what to do next.
  
  Also, you might try starting with a GRU (the LSTM overhead may not be needed).
  
  **One Idea for a starting model**
  
  Observe that turnout **does not** affect weather. So the weather forecast model will be independent of the turnout model but not vice versa.
  
  (Note: I'm using RNN to denote which ever recurrent architecture you choose)
  
  **Formulation for simultaneous prediction**:
  
  Weather
  
  $\text{RNN}\_w$ is your weather prediction model
  
  $\hat w\_t = \text{RNN}\_w(\hat w\_{t-1})$ be the predicted weather at time $t$
  
  Turnout: The idea here is that people will or will not go to the beach for various reasons (overcrowded, too hot or stormy etc). So in the beach population prediction task we use all these as **features** to predict the population at the next time-step. This reduces the problem to any one of the classical models already developed.
  
  $\text{RNN}\_p$ is your turnout (beach population) prediction model
  
  $\hat p\_t$ be the predicted turnout at time $t$
  
  $\hat c\_t=[\hat p\_t, \hat w\_t]$ is the concatenation at time t
  
  $\hat p\_{t+1} = \text{RNN}\_p(\hat c\_t)$
  
  **Formulation in the presence of weather forecast**:
  
  Simply replace $\hat w\_t$ with the true weather forecasts $\hat f\_t$.
  
  **A final warning**
  
  Unless you think your RNN is better at weather prediction than the forecasting systems I would **not** recommend using $\text{RNN}\_w$ in a production application.
  
  I hope this helps.
  
  
  ---
  
  