rui_zhang_331:

If in my dataloader, I want to get all data in a index way, which means __get_item__(self, idx) in M data loader in N different processing can work together to process different index, how can I ensure they do not process duplicate or miss process some?


I’m not sure I understand the question completely.
The DataLoader will use multiple processes and make sure that each worker gets unique indices.
Are you using another multiprocessing wrapper on top of multiple DataLoaders?
If so, what’s the reason for it? Couldn’t you increase the number of workers in a DataLoader instead?