So, does it mean I need to create a Dataset for every file in the directory? I have like 118 million train rows split in 3000 files.
Assuming there are N rows in each file (which now is  Dataset), will the ConcatDatastet read individual Datasets into memory?