Well my current “At leas it works™” solution is to just disable shuffling and hope for the best
My dataset has such a big length because i made it so i could sample at any point, so len(dataset) = len(h5dataset) - seq_length, and since i use 44100hz wav files, it creates a dataset of crazy length
One solution might be to sample only from idx*seq_length points. It limits data variety a little bit, but it might enable suffling
Ill try it out, will write back with results