I am confused about a few things here. For example, why is all of the file loaded when only a single (column?) is returned at the end? Additionally, why is data repeatedly appended to without checking if a given index has already been retrieved? If you say that all of your data cannot fit in memory, it looks like this solution will keep increasing the amount of data stored in memory without ever deleting anything. Finally, are you using this dataset directly without a DataLoader that provides more parallelism?
For the first issue if it turns out that you don’t need to read the entire file, you might want to see if you can use skip_header in the numpy params: numpy.genfromtxt — NumPy v1.20 Manual 1. In case you want to skip columns instead you could consider “transposing” your csv files offline so that you can skip rows at data loading time.