As far as I know the Dataset will be copied to the workers in case you are using multiple workers.
This also means that you cannot modify the Dataset inplace anymore.
Here is a small example:
class MyDataset(Dataset):
    def __init__(self):
        self.data = torch.ones(10)
        
    def __getitem__(self, index):
        self.data[-(index+1)] = 2.
        return self.data[index]
    
    def __len__(self):
        return len(self.data)

dataset = MyDataset()
loader = DataLoader(
    dataset,
    num_workers=0,
    shuffle=False
)
for data in loader:
    print(data)
for data in loader:
    print(data)


dataset = MyDataset()
loader = DataLoader(
    dataset,
    num_workers=1,
    shuffle=False
)
for data in loader:
    print(data)
for data in loader:
    print(data)

While the data is already modified in the second loop for num_workers=0, it stays the same for num_workers=1.
I’m not sure, how you are implementing the caching, and there could still be a valid approach.
Could you share your implementation of caching?