Hi，I was trying some preliminary models on a huge single datafile (~150G, each row represents a data point) which is not able to be loaded into memory all at once. Therefore, I wrote a data-iterator as follows for batch-wise loading&training. However, I found the loading procedure is very slow. When the batch size is set to 1024, it takes ~0.9s to load making loading data a bottleneck in the experiment. It seems that torch.utils.data.DataLoader is preferable for those datasets where each data point can be efficiently accessed and not suitable for this case.
I was wondering if anyone can provide some recommendations on improving the loading efficiency? Millions of thanks in advance.

def data_iterator(data_path, batch_size):
    print("Loading data in {}".format(data_path))
    count = 0
    X = list()
    y = list()



    with open(data_path, 'rb') as fr:
        for line in fr:
            items = line.strip().split('\t')
            label = int(items[2])
            trip = items[5]
            trip = json.loads(trip)
            for frame in trip:
                count += 1
                frame = np.asarray(frame)
                frame = np.reshape(frame,(1,200,50))
                X.append(frame)
                y.append(label)
                if count % batch_size == 0:
                    yield X, y
                    del X
                    del y
                    X = list()
                    y = list()
    if len(X)>0:
        yield X, y