Each worker will create a copy of the Dataset, so if you preload the data, your memory usage should increase a lot, especially if you are using 48 workers.
I’m wondering, why a single worker seems to be slow, since you are only slicing in-memory data.
Could you try to use torch.from_numpy(self.data_frame.iloc[idx].values).float() in your __getitem__ and profile the code again?