I am using a docker container on a server, though the server has 750Gb RAM. It means that there is over 500Gb free after loading the datasets.
via nvidia-smi, I can see that all GPUs get the data and it fails as soon as the last GPU gets its batch.
Also, I usually see that after dataparallel all GPUs get a replica of the model while currently, only the master GPU gets the model and the rest get it together with the batch (if it helps).