It likely depends on the particular bottlenecks of your system. If there is a large amount of system memory, but limited file I/O, you might want to keep the entire file in memory at all times. If there isn’t much system memory, but sufficient file I/O you could split the file. The actual file format may not matter as much as the basic loading algorithm design.