Hi,
Since my dataset is a bit large and takes a lot of time to load to RAM (But the RAM is large enougt). I’m thinking if we can load the data to RAM, when the data has been used for the first time.
That means, I want to create an empty data list in init() to save the loaded data. And data will be written into the list in get_item(). Since get_item() is called in dataloader with multiple workers, so in this case, the data list can be filled very fast.
However, this not works. I think, it is because that every worker has its own memory. So after 1 epoch, the data list will be cleaned.
Do you have any suggestions for this(like speed up loading the data in RAM)? Maybe like create the data list in the shared memory?
Thanks!