Thank you, Ditlev and wizardk, for responding.



 Ditlev_Jorgensen:

Can you pass me a link to something confirming your claim, as Im very intrigued by the idea myself, and would love to know more.


Yup. I found these very interesting. For those that might need a little more reason to click, they found that LSTM actually doesn’t have that long of a memory, and that using a set of dilated CNNs could develop an exceptionally long memory about the sequence.
arxiv.org 1 30
arxiv.org 2 19
towardsdatascience.com 40



 Ditlev_Jorgensen:

So it might be a problem using two sample rates as the CNN might have trouble when grid structure are different from sequence to sequence.


I had planned on solving the problem of different sample rates mating with the fixed-size dense layer by using an Adaptive Pool at the top of the “feature extracting” CNN. (this also is in response to your suggestion of training two CNNs). Also, I feel I should add that this is continuous data.



 Ditlev_Jorgensen:

f you loose information at 128 Hz, is it possible to resample data to 512 Hz then?


I could, but then I blow up my data set even more. Since about 1/2 of the recordings are at 128 Hz, even without padding the sequences, we’re talking a 4x increase for just those recordings.



 Ditlev_Jorgensen:

Another solution could be to simple clip the data sequences to have same length but that would throw away a lot of otherwise usefull data.


I’m almost 100% positive that throwing away 3/4 of the data would not work. Even though I’m extracting some features that I’m not so confident the network can learn, I still expect that the bulk of the learning will occur on the raw data.



 wizardk:

There are 4 ways to deal with variable size inputs:


You list mixes possible solutions for the two -different- dimensions; but the solutions for one are not applicable to to the other. The recording sequences themselves are of variable length, and the 1D main feature vector is also variable (the same size for a given sequence, but varies by recording).



 wizardk:

Why there is a “2x”?


The difference between the shortest and longest sequence (number of blocks of data) is over 3x, combined with the 4x difference between the lowest and highest sampling rate (the samples for each block of X seconds).
I still wonder if there is another solution to this. Although I would prefer not to, if necessary, I can resample the blocks to be the same size, and pad the sequences of blocks - which would actually be smaller memory-wise. However, I’m still not quite sure how to do the padding for the upper CNNs so that computations aren’t wasted on computing empty blocks, as it appears the pad/pack functions in pytorch are just for RNNs.
Aside: I was thinking about it more after I posted last night, and I realized that given that pytorch only needs to reshape the data for a time distributed network might be of use (I had used keras’ TimeDistributed wrapper around the feature extraction part that runs on every block).