Hi everyone, what is the best practice to share a massive CPU tensor over multiple processes (read-only + single machine + DDP)?
I think torch.Storage — PyTorch 1.9.0 documentation 7 with share=True suited my needs, but I can’t find a way to save storage and read it as a tensor. (same open issue 2 on Oct 29, 2019 )
I also tried to copy training data to /dev/shm (reference) and run DDP with 8 GPUs, but nothing is different. The memory usage when running with 8 GPUs is the same as before, but I tested with a single process, loading the dataset may occupy about 1 GB of memory. Am I missing something here?