Update this thread a bit,  I refactor the training code using pytorch multi processing to speed up some data processing in CPU side(in order to feed to GPU faster ), https://pytorch.org/docs/stable/notes/multiprocessing.html#multiprocessing-best-practices 3
In each processing func, I also use multi-worker data loader to speed up the data loading processing time.https://pytorch.org/docs/stable/data.html 2
I put my heaving CPU JSON parsing not in dataloader, but in main training process, and the problem seems gone, I don’t know why but anyway it seems working. but have a follow up question: suppose I have  N  processing, each has  M  dataloader worker, so total  NxM  underneath threading there.
If in my dataloader, I want to get all data in a index way, which means  __get_item__(self, idx)  in M data loader in N different processing can work together to process different index, how can I ensure they do not process duplicate or miss process some?