I have a dataset consisting of 1 large file which is larger than memory consisting of 150 millions records in csv format.
Should i split this info smaller files and treat each file length as the batch size ?
All the examples I’ve seen in tutorials refer to images. ie 1 file per test example or if using a csv load the entire file into memory first.
The examples for custom dataset classes I’ve seen are as below. len returns the entire file length and getitem returns an individual record.
I’d have thought files larger than memory would be a common issue in this time of big data ?
def __init__(self, csv_path):

def __len__(self):
    return self.data_len
    
def __getitem__(self, index):
    return self.input[index], self.labels[index]

Any advice would be greatly appreciated.
thanks