Answers.Body: <p>I would set the problem up like this:</p>
  <h3>Data</h3>
  <p>Featurize each datapoint as <code>timepoint_feature_vector = [[station_id], [categorical_features], [continuous_features], [future_timepoint, future_station_id]]</code></p>
  <p>Your training data will be a sequence of feature vectors from all stations, sorted by time. During inference, you can use the <code>[future_timepoint, future_station_id]</code> features to specify when and what station you want predictions for.</p>
  <p>Adding <code>[future_timepoint, future_station_id]</code> helps the model learn the different behaviors of the stations. Using all station data together in a stream allows the model to use data from all stations to inform the next prediction, which can help fill in sampling gaps.</p>
  <h3>Model</h3>
  <p>Use a single LSTM model. Use embeddings for <code>categorical_features</code> and <code>station_id</code>. <code>future_station_id</code> should use the same embeddings as <code>future_station_id</code></p>
  <p>Your training setup is to predict sun intensity (and any other features you are interested in) for the next datapoint in the stream.</p>
  <h3>Inference</h3>
  <p>For inference, you want a server that holds the <code>model</code>, the <code>last_hidden_state</code>, and <code>last_data_sample</code> (featurized version of the last sample reading).</p>
  <p>To do inference, you send the server <code>[future_timepoint, future_station_id]</code> values you want a prediction for. You take <code>last_data_sample</code>, fill in the values for <code>[future_timepoint, future_station_id]</code>, and run inference using <code>last_hidden_state</code>.</p>
  <p>When you get a <code>new_data_sample</code> reading, you take <code>[future_timepoint, future_station_id]</code> from <code>new_data_sample</code>, add those to <code>last_data_sample</code>, do a forward pass to generate <code>new_hidden_state</code>. You then replace <code>last_hidden_state</code> with <code>new_hidden_state</code> and <code>last_data_sample</code> with <code>new_data_sample</code>. You need to let the <code>last_hidden_state</code> lag by one because you will be using <code>last_data_sample</code> for inference.</p>
  <h3>Development</h3>
  <p>Start with just a single station and figure out what features you need to get this to work. Then roll in multiple stations.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-04T22:13:29.300Z
Answers.LastActivityDate: 2023-10-04T22:13:29.300Z
Answers.UserDisplayName: Karl
Body: <p>I am working on the following problem:
  I have ~10 weather stations in somewhat approximate areas, at some points during the day (different for each station), I get readings of various data points (like rain, wind, air pressure), as well as sun intensity.</p>
  <p>After a reading from any of the weather stations, I would like to update my prediction of the sun intensity at some point in the future, for all stations.</p>
  <p>My first thought was to build a LSTM with 10 outputs, as well as 10 bits to map which stations data I'm inserting. But I'm not sure if it would not be better to have 10 separate LSTMs that later converge within the model? That would add a lot of complexity, but neither approach is yielding great results yet. Is there some research I am unable to find that suggests a better model?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-10-02T16:26:19.517Z
Comments.Text: How long is your time series to neural network? What regularization used? What activation function used?
Comments.UserDisplayName: Cloud Cho
Tags: <deep-learning><recurrent-neural-networks><long-short-term-memory>
Title: LSTM with multiple data streams
UserDisplayName: PenguinHook
ThreadText: # LSTM with multiple data streams
  
  (Asked by: username_0 on 2023-10-01)
  
  I am working on the following problem:
  I have ~10 weather stations in somewhat approximate areas, at some points during the day (different for each station), I get readings of various data points (like rain, wind, air pressure), as well as sun intensity.
  
  After a reading from any of the weather stations, I would like to update my prediction of the sun intensity at some point in the future, for all stations.
  
  My first thought was to build a LSTM with 10 outputs, as well as 10 bits to map which stations data I'm inserting. But I'm not sure if it would not be better to have 10 separate LSTMs that later converge within the model? That would add a lot of complexity, but neither approach is yielding great results yet. Is there some research I am unable to find that suggests a better model?
  
  **Tags:** deep-learning, recurrent-neural-networks, long-short-term-memory
  
  > **Comments:**
  > **username_1**: How long is your time series to neural network? What regularization used? What activation function used?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-10-04. Score: 1
  
  I would set the problem up like this:
  
  ### Data
  
  Featurize each datapoint as `timepoint_feature_vector = [[station_id], [categorical_features], [continuous_features], [future_timepoint, future_station_id]]`
  
  Your training data will be a sequence of feature vectors from all stations, sorted by time. During inference, you can use the `[future_timepoint, future_station_id]` features to specify when and what station you want predictions for.
  
  Adding `[future_timepoint, future_station_id]` helps the model learn the different behaviors of the stations. Using all station data together in a stream allows the model to use data from all stations to inform the next prediction, which can help fill in sampling gaps.
  
  ### Model
  
  Use a single LSTM model. Use embeddings for `categorical_features` and `station_id`. `future_station_id` should use the same embeddings as `future_station_id`
  
  Your training setup is to predict sun intensity (and any other features you are interested in) for the next datapoint in the stream.
  
  ### Inference
  
  For inference, you want a server that holds the `model`, the `last_hidden_state`, and `last_data_sample` (featurized version of the last sample reading).
  
  To do inference, you send the server `[future_timepoint, future_station_id]` values you want a prediction for. You take `last_data_sample`, fill in the values for `[future_timepoint, future_station_id]`, and run inference using `last_hidden_state`.
  
  When you get a `new_data_sample` reading, you take `[future_timepoint, future_station_id]` from `new_data_sample`, add those to `last_data_sample`, do a forward pass to generate `new_hidden_state`. You then replace `last_hidden_state` with `new_hidden_state` and `last_data_sample` with `new_data_sample`. You need to let the `last_hidden_state` lag by one because you will be using `last_data_sample` for inference.
  
  ### Development
  
  Start with just a single station and figure out what features you need to get this to work. Then roll in multiple stations.
  
  
  ---
  
  