Will try, Thanks, a little bit hard to increase segment, however, I tested with smaller file size and number of files, still same problem, 4096 segment looks definitely enough for me, and also my problem happens almost right after dataloader started, so I don’t believe the data is filling to shm enough yet.
I am trying to use multi processing not only on data loader, but general training program also, so training will have N processing, each processing will have M dataloader processing, total NxM threads/process, but I am not sure whether all independent processing’s dataloader can coordinate to read correct data index