Thank you. I tried np.memmap and that works!
I also tried to make it the perhaps more standard way of the tutorial you linked which I checked before. Is this the most common way of loading data when having large datasets, that you have all the images together with an annotation file like a csv file that you can you load into memory? This is the result I got when trying to follow that tutorial but it still seems I might be missing something, getitem feels like there should be a cleaner way of doing the same thing. Perhaps you could give it a quick look and see if I’m on the right path
class my_Dataset(Dataset):
    # Characterizes a dataset for PyTorch
    def __init__(self, root_dir, csv_file, transform=None):
        ''' Args:
                csv_file (string): Path to csv file with annotations.
                root_dir (string): Directory with all the images.
                transform (optional): transforms to be applied on a sample.
        '''
        self.root_dir = root_dir
        self.annotations = pd.read_csv(csv_file + '.csv')
        self.transform = transform

    def __len__(self):
        return len(self.annotations)

    def __getitem__(self, index):
        img_name = os.path.join(self.root_dir, self.annotations.iloc[index].name)
        image = io.imread(img_name)
        image = np.moveaxis(np.asarray(image), [2], [0])
        image = torch.from_numpy(image)
        y_label = self.annotations.iloc[index,:].values
        y_label = torch.from_numpy(self.annotations.iloc[index,:].values)

        if self.transform:
            image = self.transform(image)

        return image, y_label


I’ve seen you answer a lot of questions and I just want to say that I appreciate you helping us all out. It really means a lot.