Hi @ptrblck thanks for the code sample.
I wanted to ask a related question.
My dataset object does not contain a straightforward tensor that will be read in chunks by dataloader workers.
In fact, I have a custom python object which performs the role of a search tree and everytime a worker triggers the __getitem__ method of the dataset, the search tree is used to get the appropriate data.
In general, how could I ensure that an arbitrary python object (neither pytorch tensor nor np array) is not copied when I used more than one worker in the dataloader? I have made sure I never write to the structure when loading data but I still get memory overhead due to copying.
Thanks for your help!