Running on 4+ gpus on the same machine, and I’m trying to save time and memory by just caching all of the Dataset items once and sharing that list among all of the Dataloader workers for all of the currently training models on the same machine.
I’ve been able to get a Manager.list that is shared among all of the Dataloader workers for one model just fine. However, I’m not sure how to share a pointer to this list with the other models that are started later.