Hi, Thanks for the reply.
I had a go at the suggestions you made. For starters, in __init__ , I’m unsure of whether I can open the hdf5 file and calculate the length.
In _getitem_ , I reference the archive file and return the indexed values. I don’t get an OS B-tree error anymore, however, I’m not sure if this the most optimal way of retrieiving the data. Here is my newer attempt:
import torch.multiprocessing as mp
mp.set_start_method('fork') 

from torch.utils import data
import h5py

class Features_Dataset(data.Dataset):
    def __init__(self, file_path, phase):
        self.file_path = file_path
        self.archive = None
        self.phase = phase 
        with h5py.File(file_path, 'r', libver='latest', swmr=True) as f:
           self.length = len(f[(self.phase) + '_labels'])

    def _get_archive(self):
        if self.archive is None:
            self.archive = h5py.File(self.file_path, 'r', libver='latest', swmr=True)
            assert self.archive.swmr_mode
        return self.archive


    def __getitem__(self, index):
        archive = self._get_archive()
        label = archive[str(self.phase) + '_labels']
        datum = archive[str(self.phase) + '_all_arrays']
        path = archive[str(self.phase) + '_img_paths']

        return datum[index], label[index], path[index]

    def __len__(self):
        return self.length

    def close(self):
        self.archive.close()

if __name__ == '__main__':
    train_dataset = Features_Dataset(file_path= "featuresdata/train.hdf5", phase= 'train')
    trainloader = data.DataLoader(train_dataset, num_workers=8, batch_size=1)
    print(len(trainloader))
    for i, (data, label, path) in enumerate(trainloader):
        print(path)



Additionally, you are right about the small network. I am using a small autoencoder, where I am sending values from the hdf5 file to this network:
import torch
import torch.nn as nn

class AutoEncoder(nn.Module):
    def __init__(self, n_embedded):
        super(AutoEncoder, self).__init__()
        self.encoder = nn.Sequential(
            nn.Linear(6144, n_embedded))
        self.decoder = nn.Sequential(nn.Linear(n_embedded, 6144))
       
    def forward(self, x):
        encoded = self.encoder(x)
        decoded = self.decoder(encoded)
        return encoded, decoded


I also have tensorboard set up where I am supposed to see my training losses every 100 mini-batches. It appears that nothing is recorded. Furthermore, when I set pin_memory to True in my dataloader, nothing is being written at all. I am using 4 CPUS and 2 GPUs to load and train my networks, if that matters…