Hello everyone,
I am facing some memory issues running my model on multiple GPUs with DDP. I want to report to you the experiments I made to understand the memory utilization of the combination of workers + DDP. In all the experiments I will report in this post I will use always the same model, so the size of the model is always the same.
I have 4 GPUs with 16GB each. First of all, I used only one to understand the highest batch that fits on it. The maximum batch size is 64 (I started to have CUDA OOM with 128). Then, in order to increase the batch size, I started to use DDP with batches of 64 for each process (1 process per GPU). Using 2 GPUs with a total batch of 128 caused CUDA OOM. I have read online that the DDP uses part of the GPU memory to store the communication bucket. So I assume that this communication bucket does not fit into memory. I started to decrease the per-GPU batch size to 60, then 55, and finally to 50 (resulting in a total batch size of 100 on 2 GPUs). Unfortunately, none of these batch sizes worked, For curiosity, I started to analyze the memory usage of 1 single GPUs with a batch of 1. The total memory usage is 3.4G. I then tried to use DDP with a batch of 2 (1 per-GPU) resulting in 3.9G. This experiment demonstrates the overhead made by DDP. Finally, I tried to increase the number of workers and I have found that the final memory usage on GPU increases. This last founding shocked me because I thought that the workers simply increase the RAM usage, not GPU.
More specifically, whenever I use DDP I have several processes per GPU. The first process I see is the one performing the training (3.9G), then there are N_GPUS-1 processes that I suppose are in charge of keeping the communication with the other processes opened (0G each), and finally, the other ones should be the workers (0.6G each). Thus the memory usage on each process is 3.9G + num_workers*0.6G.
It’s not clear to me why the workers have a memory footprint on the GPU. Can someone help me to understand?