Hi
I have implemented a custom dataset class where images are retrieved based on numerous object attributes. This, unfortunately, doesn’t work when I employ multiple workers. I am not at all certain that this is the right way of implementing this type of functionality. I’d appreciate any help. Here is some pseudocode:
loader = dataloader(custom_dataset)
for X1, y, indexes in enumerate(loader): # where indexes are the indexes of the loaded batch
    pass X1 into a model and get some values V1
    update custom_dataset with V1 for current indexes
    retrieve X2 using __getitem__(indexes) 
    pass X2 into a model and get some values V2
    update custom_dataset with V2 for current indexes
    retrieve X3 using __getitem__(indexes)
    pass X3 into a classifier, get y hat, compute the loss, do a backpropagation

Unfortunately, I cannot preprocess the data a priori and hence there needs to be sequential data retrieval from the same images. If there was some way of passing the values V1, V2, V3 into getitem, I think multiprocessing would work. Let me know if anything is unclear! Thank you!