Should add that I am using a custom IteratableDataset that also implements indexing.
Unfortunately I cannot fix this by joining the raw files and loading them as one since the data is interpreted as sequences and joining them would create invalid sequences right where two files meet.