I’m working with a dataset of tens of thousands of high-frequency time series, meaning each segment of one second of data is ~50 MB. I’m trying to avoid downsampling or other lossy strategies if at all possible. This obviously presents significant computational challenges, one of which is that any copying of the data incurs a significant time penalty.
Say the data is a 2D array where the rows are samples and the columns are the time series, and I have it all loaded into RAM (in reality, I have much more data than can fit in even regular memory, but let’s leave that aside for now).
In theory, this should mean I can use views into the full array to work with subsets of it rather than copying data, at least when they’re contiguous. And although I have much less GPU memory than regular memory, the same general principle should apply if the dataset is on the GPU.
One place where I can’t think of a way to avoid copying is in constructing minibatches of segments of data, where each segment is a continuous stretch of, say, one second of contiguous samples from every time series – i.e. a slice of the rows of the original array – but the segments making up the batch aren’t necessarily continuous.
Is there any way to do something like constructing a tensor for a batch where the first (batch size) dimension is actually just indexing views of row slices of another tensor?
Or, perhaps there wouldn’t be much benefit to using minibatches anyway, and I should just stick with pure (single-sample) stochastic gradient descent. It’s also entirely possible that I’m overly focused on this particular issue when others would make it moot (e.g. the computation will be much slower than the copying). At the moment I’m not using DataLoader, Dataset, or any other PyTorch data management tools, but if they would address this issue I’d be happy to hear it. Any thoughts on this subject would be much appreciated.