LSTM data preparation with multiple independent observation runs
I'm struggling to wrap my head around how to correctly deal with multiple independent time series observations for training an LSTM. If I simply concatenate my observation runs, then there will be an incorrect disjoint between the last value and first value of adjacent sequences.
By way of hypothetical example:
Say I'm doing some sort of semi-stochastic time series experiment. Let's say every day from 9am-10am I measure sound levels once per second at an intersection and record the number of cars driving by at that moment. And say I do this for 10 days.
So my data from each day is three columns: (time (sec), dB, cars)
And I have 10 of these datasets; one from each day I took measurements. Call them obs_day01, obs_day02, ..., obs_day10
So far so good.
The time interval is always the same, so I can remove that from my data, leaving me with 10 independent observation series of format (dB, cars).
A first thought would be to concatenate these into one long series and then use TimeseriesGenerator in Keras. But this won't work, because the boundary of every concatenation is uncorrelated. 650
i.e. The observation at 09:59:59 in obs_day01 is uncorrelated to the observation at 09:00:00 in obs_day02. So if I concatenate my datasets, the LSTM will learn a pattern at those edges that isn't there.
So a second thought becomes to write a custom generator to pass to model.fit() that generates its batches of samples by drawing sub-sequences from all 10 sets of observations, but ensuring that it never generates a training sequence that "crosses" the 09:59:59 - 09:00:00 boundaries. Something like:
def customGenerator(args): # Pseudocode
while True:
# create batch_size subsequences
# of seq_len, sampling from all 10
# observation sets, but ensuring that
# individual sequences come only from
# a single day's observations
So it seems like maybe that would work, but
1) I'm realizing I still don't understand how to approach the problem of multiple sequence samples with training LSTMs, and
2) This doesn't seem like it should be as complicated as I'm making it. In standard (non-recurrent) deep neural networks, we always train with many samples of the same "type," so surely RNNs are no different, which means there's something here I'm not grokking...
Hope that makes sense. Guidance much appreciated... 650