I’m trying to train a model on 8 GPUs at once (on the same machine), using DistributedDataParallel. My issue is that this ends up using a lot of memory and I have to “cripple” my dataloader a bit in order even make it fit on the machine’s 64 GB of RAM.
The reason seems to be a combination of two things. First, my dataloader upon being initialized loads the entire dataset (about 3GB total) into RAM, so that loading batches later will be very fast. Second, I can see that a separate python process is created for each GPU that I train on (even with num_workers=0 in the dataloader), and each of these processes uses a substantial amount of RAM, presumably because each process loads its own copy of the entire dataset into RAM. This scales linearly with the number of GPUs I use.
I’m not sure the above fully explains the problem as the data is only 3GB, and yet each process uses about 6GB of RAM - unless perhaps the data is stored in RAM in a more bulky (e.g. lower-precision) format than on disk? Or perhaps the remaining RAM is because of other things being stored in RAM, like model weights or gradients (but then I thought those should live in GPU memory)?
I guess my main question is: is this normal / inevitable behavior? If the problem is indeed that 8 copies of the dataset end up getting stored in RAM, is there some way to avoid this and load it into RAM only once? Or is this issue caused by something else?