Hi, sorry for the late replay.
You are not using mmap “properly”.
mmap should be defined in the init functionn (not in getittem).
Once you define it you can treat it as a normal array.
When you try to retrieve the data it will read it from disk.
Defining the mmap instance in getitem makes no sense to me.
The problem with shuffling is it takes lot of memory to generate indices. Besides, you will encounter this problem of reading contiguous blocks of memory. It’s indeed way faster than reading separate blocks.