Alright, now i am sampling at idx*seq_length points, and shuffling works nicely. Lesson is: giant dataset will be giant at least somewhere: if not in ram used to store it, then in ram used to shuffle it, or in time you will spend debugging model trained on unshuffled dataset